REVIEW 3 major objections 6 minor 13 references
Likelihood and Deep Learning Analysis of the electron neutrino event sample at Intermediate Water Cherenkov Detector (IWCD) of the Hyper-Kamiokande experiment
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A deep-learning event selection for the Intermediate Water Cherenkov Detector raises simulated electron-neutrino sample purity from 51.1% to 61.5% while improving efficiency from 69.5% to 78.2% compared to the likelihood-based fiTQun…
desk verdict A plausible but in-sample ML improvement for IWCD electron neutrino selection; the reported purity/efficiency gains are likely optimistic until validated on a fresh sample. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the trained softmax output of an 18-layer residual convolutional network (ResNet-18). The network converts raw PMT hit patterns into four class probabilities $P(e)$, $P(\mu)$, $P(\gamma)$, and $P(\pi^0)$; the selection then applies manually chosen two-dimensional cuts on these probabilities versus reconstructed momentum, layered on the same fiducial-volume and kinematic cuts used in the fiTQun analysis. The baseline against which the claim is measured is fiTQun, a maximum-likelihood event reconstruction whose likelihood ratios are used for particle identification.
What would settle it
Train the identical network and apply the identical softmax cut thresholds to a freshly generated independent IWCD Monte Carlo sample, then recompute purity and efficiency; if the numbers fall materially below 61.5% and 78.2%, the reported gain is overfit to the tuning sample. A simpler check is to split the existing simulated beam sample in half, tune on one half and evaluate on the other.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a ResNet-18 convolutional network trained on four particle classes ($e^-$, $\mu^-$, $\gamma$, $\pi^0$) yields softmax probabilities that, after manual cuts tuned against reconstructed momentum, select $\nu_e$CC0$\pi$ events at higher purity and efficiency than fiTQun's likelihood ratios on the same simulated beam sample. The ML selection retained 2856 signal events versus 2535 for fiTQun, achieved 61.5% purity and 78.2% efficiency, and suppressed NC$\pi^0$ and NC$\gamma$ backgrounds more strongly, at the cost of admitting more $\nu_e$CC-other and $\bar{\nu}_e$ events. The authors conclude that with further development, more complex and automated machine-learning cuts should improve the sample beyond what is reported.
Load-bearing premise
The cuts producing the 61.5% purity and 78.2% efficiency were tuned by eye on the same simulated sample from which the final numbers are drawn, so the result assumes that sample is a faithful, unbiased stand-in for real IWCD data and that the tuning did not overfit its statistical fluctuations.
Editorial extensions
If this is right
- A convolutional-network PID can serve as the $\nu_e$ event selection for IWCD, replacing likelihood-based cuts with a single pass through a trained classifier.
- The higher purity and efficiency reduce the statistical and systematic footprint of the $\nu_e$ cross-section measurement, which is one of the inputs to the CP-violation sensitivity of the Hyper-Kamiokande program.
- Because NC$\pi^0$ and NC$\gamma$ contamination is the hardest part of the likelihood selection, the ML result shows these backgrounds can be suppressed substantially without losing signal.
- The authors expect further development of more complex and automated machine-learning cuts to push the sample's purity and efficiency beyond the reported values.
Reading between the lines
- The reported numbers are in-sample estimates because the softmax cuts were manually tuned on the simulated beam sample used to evaluate them; an independent test sample would likely show a smaller (though possibly still positive) improvement.
- The network was trained only on single-particle particle-gun events, not on full neutrino interaction topologies; retraining or fine-tuning on beam-like events with multiple rings could improve rejection of asymmetric $\pi^0$ decays further.
- The same softmax-plus-cut strategy could transfer to Hyper-K's far detector, but the different PMT geometry and granularity make that transfer an open question.
- Because the ML selection increases the $\nu_e$CC-other and $\bar{\nu}_e$ backgrounds, the net benefit for a cross-section measurement depends on how well those backgrounds are constrained independently.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This ICHEP 2024 proceedings paper compares two event-selection methods for electron-neutrino charged-current quasi-elastic (nu_e CC0pi) events in the Intermediate Water Cherenkov Detector (IWCD) of Hyper-Kamiokande, using simulated data from NEUT, WCSim, and fiTQun. The fiTQun-based analysis applies fiducial-volume, kinematic, and likelihood-ratio cuts and reports a purity of 51.1% and efficiency of 69.5%. The authors then train a ResNet-18 CNN within the WatChMaL framework on particle-gun simulated events and apply it to the same simulated beam sample; after manually tuning three softmax-probability discriminators, they report an improved purity of 61.5% and efficiency of 78.2%. The paper concludes that the ML-based selection outperforms fiTQun and expects further gains with automated cut optimization.
Significance. If the reported improvement is robust, it is a useful result for Hyper-Kamiokande's IWCD physics program: better nu_e event selection would reduce backgrounds for cross-section measurements and, ultimately, for CP-violation sensitivity. The paper provides concrete simulated event counts and a direct head-to-head comparison with an established likelihood-based reconstruction, which is valuable to the community. The main weakness is statistical: the ML cut thresholds are manually tuned on the same simulated beam sample used to quote the final purity and efficiency, so the headline numbers are in-sample estimates. A proper validation on a held-out sample, or cross-validation, is needed before the central claim can be accepted as stated.
major comments (3)
- [Section 3, Figure 4, Table 1] The central claim that the ResNet-18 selection achieves 61.5% purity and 78.2% efficiency is based on in-sample evaluation. The paper states that the three discriminators were manually tuned 'based on the distribution of signal and background in the histograms' (Figure 4), and that the same 'test dataset' is used for ML model evaluation. The final purity and efficiency in Table 1 are then quoted from that same sample after applying the tuned cuts. No independent validation split, cross-validation, or hold-out set is described. This is a load-bearing issue because manual threshold tuning can exploit statistical fluctuations in the simulated sample, making the reported improvement over fiTQun optimistic. Please evaluate the selected sample on events that were not used for tuning the ML cuts, and report the purity/efficiency with statistical uncertainties.
- [Sections 2.1 and 3, Table 1] The comparison between fiTQun and ML is not made on equal footing. The fiTQun cut lines are optimized using a predefined figure of merit, FOM = S/sqrt(S+B), whereas the ML softmax cuts are manually tuned to improve both purity and efficiency without an explicit, common objective. This asymmetry means that the ML result includes an additional hand-tuning step that can absorb favorable statistical fluctuations. A fairer comparison would either optimize both methods with the same FOM on a training sample and evaluate both on a separate test sample, or use a blinded analysis strategy.
- [Table 1] The central table is not self-explanatory as printed. The header appears to list more categories than can be unambiguously matched to the numeric entries, and the sum of the listed counts does not directly reproduce the denominators implied by the quoted purities (e.g., for the ML row, the listed categories sum to 5362 events, while the quoted purity of 61.5% corresponds to a denominator of about 4642 events). Please restate Table 1 with explicit column definitions, a clear 'total selected' row/column, and the exact formula used for the purity denominator, so that the results are reproducible.
minor comments (6)
- [Section 3] The terminology 'test dataset' is confusing: the same sample is used both for manually tuning the discriminators and for reporting the final numbers. Please use consistent terms such as training/validation/test, or describe the split explicitly.
- [Table 1] No statistical or systematic uncertainties are reported for the event counts, purity, or efficiency. At minimum, Poisson or binomial errors on the MC counts should be given, and the absence of detector/systematic uncertainties should be stated as a limitation.
- [Section 3] The input representation for the ResNet-18 model is not described: what exactly is fed to the network (mPMT hit images, charge/time channels, etc.)? Also, details of the train/validation split used during the 20-epoch training are omitted.
- [Section 2] The simulation chain is not fully referenced: NEUT and WCSim are mentioned in the text but no citations are provided. Please add the appropriate references.
- [Section 2.1] The statement that fiTQun processes 'at most 1 event per minute' is a quantitative runtime claim without supporting measurements or hardware specifications. If computational cost is part of the motivation, provide a benchmark on the same hardware for both methods.
- [General] Some figure references and caption details are incomplete in the extracted text (e.g., axis labels, color scales, and exact definitions of 'Total NC' versus 'NC pi0' and 'NC gamma'). Please ensure the final version includes clear, self-contained figure and table captions.
Circularity Check
Reported ML purity/efficiency are in-sample estimates because the three discriminators were manually tuned on the same beam sample used for Table 1; the comparison with fiTQun may be optimistic.
-
fitted input called prediction
[Section 3, Figure 4, Table 1]
"Based on the distribution of signal and background in the histograms, we manually tuned three discriminators across P(μ), P(π0), and P(e) to improve both the purity and efficiency. After applying all these ML cuts sequentially along with the basic FV and kinematic variable cuts (as defined for the fiTQun-based analysis), the sample’s purity improved to 61.5%, with an increased efficiency of 78.2% (Table 1)."
The histograms in Figure 4 are generated from the same test dataset that the paper says is used for ML model evaluation and that is counted in Table 1. The three discriminators are thus manually fitted to the very sample whose purity (61.5%) and efficiency (78.2%) are then reported as the result. This is an in-sample, post-tuning performance estimate: the thresholds are chosen by looking at signal and background distributions in that sample, so the quoted numbers are not an unbiased prediction for an unseen dataset. The fiTQun cuts were also optimized on the same sample using FOM = S/√(S+B), so the comparison is between two in-sample estimates; the ML side has an additional hand-tuning step that can absorb statistical fluctuations.
full rationale
The paper’s central comparison (ML 61.5%/78.2% vs fiTQun 51.1%/69.5%) is the only quantitative prediction in the paper. It is not a mathematical derivation, so there is no equation-level circularity. However, the ML thresholds are manually tuned on the same simulated beam sample whose event counts appear in Table 1, making the headline numbers trained-sample estimates. The underlying CNN itself was trained on independent particle-gun events (e−, μ−, γ, π0), and the ROC comparison in Figure 3 provides independent evidence for the ML approach; that is why this is partial circularity rather than a complete reduction to input. The paper presents no cross-validation, independent validation sample, or uncertainty on the efficiency and purity numbers. The fiTQun result is also in-sample (FOM optimized on the same sample), but the ML result includes an extra manual tuning stage, so the reported margin should not be taken at face value. Score 4 reflects one significant in-sample-fitting step while the core ML classifier retains independent content.
Assumptions & free parameters
free parameters (4)
- ML softmax cut thresholds for P(e), P(mu), P(pi0) =
not reported
- fiTQun likelihood-ratio cut lines =
not reported
- ResNet-18 network weights =
trained, not reported
- Fiducial volume and kinematic cut values =
Dwall > 100 cm, Towall > 100 cm, momentum > 100 MeV/c
assumptions (4)
- domain assumption NEUT neutrino interaction generator correctly models nu_e/nu_mu CC and NC processes for IWCD
- domain assumption WCSim detector simulation accurately reproduces the IWCD mPMT response
- domain assumption The fiTQun likelihood function in Eq. (1) is a valid model of PMT hit probabilities for particle hypotheses
- domain assumption The ResNet-18 architecture and WatChMaL framework are appropriate and correctly implemented
Cite this review
Pith. "Pith review of Likelihood and Deep Learning Analysis of the electron neutrino event sample at Intermediate Water Cherenkov Detector (IWCD) of the Hyper-Kamiokande experiment." pith.science (2026). https://pith.science/paper/Q7EAG5X4
@misc{pith2026241109562,
author = {Pith},
title = {Pith review of: Likelihood and Deep Learning Analysis of the electron neutrino event sample at Intermediate Water Cherenkov Detector (IWCD) of the Hyper-Kamiokande experiment},
year = {2026},
howpublished = {\url{https://pith.science/paper/Q7EAG5X4}},
note = {Machine review of arXiv:2411.09562}
}
abstract
Hyper-Kamiokande (Hyper-K) is a next-generation long baseline neutrino experiment. One of its primary physics goals is to measure neutrino oscillation parameters precisely, including the Dirac CP violating phase. As conventional $\nu_{\mu}$ beam generates from the J-PARC neutrino baseline contains only 1.5$\%$ of $\nu_{e}$ interaction of total, it is challenging to measure $\nu_{e}/\bar{\nu}_{e}$ scattering cross-section on nuclei. To reduce these systematic uncertainties, IWCD will be built to study neutrino interaction rates with higher precision. Simulated data comprise $\nu_{e}CC0\pi$ as the main signal with NC$\pi^{0}$ and $\nu_{\mu}CC$ are major background events. To reduce the backgrounds initially, a log-likelihood-based reconstruction algorithm to select candidate events was used. However, this method sometimes struggles to distinguish $\pi^{0}$ events properly from electron-like events. Thus, a Machine Learning-based framework has been developed and implemented to enhance the purity and efficiency of $\nu_{e}$ events.
Figures
Reference graph
Works this paper leans on
-
[1]
K Abe, C Bronner, Y Haga, Y Hayato, M Ikeda, K Iyogi, J Kameda, Y Kato, Y Kishimoto, Ll Marti, et al. Atmospheric neutrino oscillation analysis with external constraints in super- kamiokande i-iv.Physical Review D, 97(7):072001, 2018
work page 2018
-
[2]
Hyper-Kamiokande Proto-Collaboration, :, K. Abe, Ke. Abe, H. Aihara, A. Aimi, R. Akutsu, C. Andreopoulos, I. Anghel, et al. Hyper-kamiokande design report, 2018. 5 IWCD electron neutrino event selection for Hyper-K experiment T. Mondal
work page 2018
-
[3]
Evidence for an oscillatory signature in atmospheric neutrino oscillations
Y Ashie, J Hosaka, K Ishihara, Y Itow, J Kameda, Y Koshio, A Minamino, C Mitsuda, M Miura, S Moriyama, et al. Evidence for an oscillatory signature in atmospheric neutrino oscillations. Physical review letters, 93(10):101801, 2004
work page 2004
-
[4]
Cpviolationandneutrinooscillations
HiroshiNunokawa,StephenParke,andJoséW.F.Valle. Cpviolationandneutrinooscillations. Progress in Particle and Nuclear Physics, 60(2):338–402, Apr 2008
work page 2008
-
[5]
An intermediate water cherenkov detector at j-parc
Mark Scott. An intermediate water cherenkov detector at j-parc. InProceedings of the 10th InternationalWorkshoponNeutrino-NucleusInteractionsinFew-GeVRegion(NuInt15) ,page 010039, 2016
work page 2016
-
[6]
RB Patterson, EM Laird, Y Liu, PD Meyers, I Stancu, and HA Tanaka. The extended-track event reconstruction for miniboone.Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 608(1):206– 224, 2009
work page 2009
-
[7]
Andrew D Missert, T2K Collaboration, et al. Improving the t2k oscillation analysis with fitqun: a new maximum-likelihood event reconstruction for super-kamiokande. InJournal of Physics: Conference Series, volume 888, page 012066. IOP Publishing, 2017
work page 2017
-
[8]
M Jiang, K Abe, C Bronner, Y Hayato, M Ikeda, K Iyogi, J Kameda, Y Kato, Y Kishimoto, LlMarti,etal. Atmosphericneutrinooscillationanalysiswithimprovedeventreconstruction in super-kamiokande iv.Progress of Theoretical and Experimental Physics, 2019(5):053F01, 2019
work page 2019
Show all 13 references
-
[9]
Deep learning.nature, 521(7553):436– 444, 2015
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.nature, 521(7553):436– 444, 2015
2015
-
[10]
AdvancesinsimulationandreconstructionforHyper-Kamiokande
NickProuse. AdvancesinsimulationandreconstructionforHyper-Kamiokande. In Proceed- ings of 40th International Conference on High Energy physics — PoS(ICHEP2020), volume 390, page 919, 2021
2021
-
[11]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016
2016
-
[12]
Watchmal: Github repository.https://github.com/WatChMaL/WatChMaL
WatChMaL. Watchmal: Github repository.https://github.com/WatChMaL/WatChMaL. GitHub repository
-
[13]
Machine learning techniques to enhance event reconstruction in water cherenkov detectors
Nicholas Prouse, Patrick de Perio, and Wojciech Fedorko. Machine learning techniques to enhance event reconstruction in water cherenkov detectors. InPhysical Sciences Forum, volume 8, page 63. MDPI, 2023. 6
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.