REVIEW 3 major objections 6 minor 34 references
Robust resonant anomaly detection with NPLM
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that in-sample density-ratio learning with NPLM detects rare resonant signals more reliably than BDT-based classifiers, and that a hyper-test over selection thresholds removes most of the variance from hyperparameter…
desk verdict Solid integration of NPLM into resonant bump hunts, with honest caveats; the main advertised regime (imperfect background) is untested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the NPLM training loss $L_{\mathrm{NPLM}}[f_w] = \sum_{x \in R} w_x(e^{f_w}-1) - \sum_{x \in D} f_w$, where $R$ is the reference background sample and $D$ is the data sample; minimizing it produces a function $f_w$ that approximates the log-density ratio $\log n(x|D)/n(x|R)$, and the test statistic is $t_{\mathrm{NPLM}} = -2\min_w L_{\mathrm{NPLM}}[f_w]$. Because the classifier is trained and evaluated on the same full dataset, no events are held out, which preserves sensitivity to rare signal events. The end-to-end version feeds all six variables, including the resonant mass, into this loss and directly produces a Neyman-Pearson test statistic with no selection threshold. The classifier version instead computes a cut-and-count statistic after a threshold, and the paper stabilizes it with a hyper-test that takes the minimum p-value over several threshold choices, borrowing the multiple-testing idea from the recent NPLM literature.
What would settle it
Run the same LHCO benchmark with a background template produced by fitting the sidebands (rather than using the true background) and compare median Z-scores at the lowest signal fraction, $N(S)/N(B)=8.2\times10^{-4}$; the paper's central claim fails if the NPLM median Z-score falls below the BDT hyper-test median.
Extended reading notes
Core claim
The central claim, stated on the paper's own terms, is that NPLM-based strategies outperform BDT-based classifiers in detection power at low signal injection while substantially reducing epistemic variance due to hyperparameter choices. The paper supports this with controlled numerical experiments on two benchmarks: the LHCO dijet dataset, where the signal is a narrow resonance, and the RODEM dataset, where the signal is a flat excess in the invariant mass. In the low-injection regime, the median Z-score of the NPLM-classifier with a hyper-test is up to a factor of three higher than the BDT counterpart, and the spread across hyperparameter choices is smaller. The authors attribute the gain to the in-sample nature of the NPLM test, which uses the full dataset, in contrast to the k-fold out-of-sample evaluation used for BDTs. The discovery is therefore a concrete demonstration that in-sample likelihood-ratio estimation is a useful alternative to standard classification for rare-signal searches.
Load-bearing premise
The numerical comparisons assume a perfect background template in the signal region, so the reference sample exactly represents the no-signal distribution; if real templates carry substantial errors, the claimed advantage of in-sample training is not guaranteed.
Editorial extensions
If this is right
- Searches for rare resonances should gain discovery power if NPLM replaces BDT classifiers in the anomaly-selection stage.
- End-to-end NPLM removes the selection-threshold hyperparameter entirely, making the analysis less dependent on analyst choices.
- The hyper-test over thresholds gives a stable way to combine classifier scores when the background template is imperfect.
- The in-sample principle suggests that other full-data density-ratio learners could beat k-fold classifiers in low-signal regimes.
Reading between the lines
- Because the paper's numerical comparisons use a perfect background template, a natural next test is to replace it with a sideband-derived template and see whether NPLM's low-injection advantage survives real background uncertainty.
- A testable prediction of the paper's explanation is that template errors hurt NPLM more than BDT pipelines, since the NPLM loss assumes the reference is the true no-signal density.
- The hyper-test idea could be extended to jointly vary BDT hyperparameters as well as thresholds, which the paper itself leaves for future work.
- Applying NPLM to a full LHC-style analysis with realistic statistical and systematic uncertainties would show whether the factor-of-three median Z-score improvement translates outside the idealized setting.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares NPLM-based resonant anomaly detection strategies with the standard BDT-based CWoLa approach on the LHCO and RODEM benchmark datasets. Two NPLM use cases are studied: (1) an end-to-end NPLM test over all six input features, applied when an accurate background template is available, and (2) an NPLM-based classifier used to select events in the signal region, followed by a cut-and-count test, including a hyper-test over several selection thresholds. Detection power is reported as power curves and median Z-scores at three signal-injection levels, with the central claims that NPLM outperforms BDTs at low signal injection and that NPLM reduces hyperparameter-induced epistemic variance.
Significance. The paper addresses a timely and practically relevant question: how to make resonant anomaly detection more sensitive to rare signals and more stable under hyperparameter choices. Its strengths are the use of public, independent benchmark datasets (LHCO and RODEM), empirical calibration via signal-free pseudo-experiments, a transparent hyperparameter scan on the BDT side, and a multiple-testing procedure over selection thresholds that reduces variance. The numerical evidence is internally consistent in the idealized perfect-template setting. However, the main comparison for the NPLM-classifier, which is motivated for the case of absent or imperfect background modelling, is carried out only under a perfect background template; the paper does not test how either method behaves under template misspecification. The variance comparison is also not fully symmetric in the hyperparameter spaces considered. If the missing misspecification study can be added and the variance claim appropriately qualified, the paper will be a solid contribution to the anomaly-detection literature.
major comments (3)
- [§3.1, §4.2, §4.3] The NPLM-classifier is introduced in Section 3.1 for use 'in absence of a good background modelling,' and Section 5 repeats that the selection-plus-calibration strategy is the appropriate one when the template is not accurately known. Yet all numerical experiments for this approach assume the idealized setting in which the template R exactly reproduces the background distribution in the signal region, as stated in Section 4.2: 'R in this work pertains to the idealised setting.' This is a load-bearing gap: with a biased template, the in-sample NPLM fit will absorb the template bias into f_w(x), the cut-and-count statistic in Eq. (7) will be shifted even under the null hypothesis, and calibration pseudo-experiments generated from the same biased R will not yield valid p-values. The reported low-signal advantage in Figs. 3 and 5 and Table 1 may therefore not transfer to the intended application. Please repeat the central comparison under at least one realistic misspecification, for example a smooth sideband fit with a shape bias in m_JJ or a template built from a shifted signal-region definition, and report null-calibration and power for both NPLM-classifier and BDT-classifier under the same misspecified R. If such a study cannot be included, the claims in Section 5 for approach (2) should be substantially weakened.
- [§4.4, Table 1] The claimed reduction in epistemic variance is not an apples-to-apples comparison. For BDTs, the hyper-test is applied only over the selection threshold thr, while the NPLM hyper-test inherits the multiple-testing procedure over the kernel width sigma from Ref. [23] and reports the remaining spread over the other NPLM hyperparameters. Table 1 and Fig. 5 therefore compare a BDT variance that includes sensitivity to nleaf and lambda but not to the hyper-test over those parameters, with an NPLM variance that has already been partially reduced by multiple testing over sigma. The paper itself acknowledges in Section 4.4 that extending the BDT hyper-test to multiple BDT hyperparameters is left to future work. Please either implement that extension, or restrict the claim to a statement about the specific hyperparameter sets used here and state explicitly how many models contribute to each shaded band.
- [§4.3, Table 1] The central low-injection advantage rests on small absolute differences in median Z-score: at N(S)/N(R) = 8.2 x 10^-4, Table 1 reports 0.30 +/- 0.04 for NPLM 5D + hyper-tcc versus 0.17 +/- 0.04 for BDT 5D + hyper-tcc. The paper does not state the number of pseudo-experiments used in Figs. 1-5 or the correlation structure between the two procedures, so it is unclear whether the difference is statistically significant rather than a fluctuation in the toy ensemble. Please report the number of toys, and provide paired bootstrap confidence intervals or an equivalent measure for the power curves and for the median Z-scores in Table 1.
minor comments (6)
- [§4.2, Eq. (8)] The definition of Z_alpha in Eq. (8) is phrased as the quantile of the normal distribution 'at the alpha complement to 1'; please write it explicitly as z_alpha = Phi^{-1}(1 - alpha) and define what is plotted on the horizontal axis of the power curves.
- [Table 1 and captions] The table caption says 'average Z-score' while the text repeatedly says 'median Z-score' and the main text says 'median Z-score among different values of the tunable hyperparameters.' Please make this consistent, and clarify whether the reported uncertainty is the standard deviation of the median or the standard error of the mean.
- [Figs. 3-5] The captions use 'standard deviation' in some places and 'standard error' in others for the shaded bands; please choose one consistent definition and state it in a common caption note.
- [§4.2, Eq. (7)] Equation (7) subtracts the number of selected template events from the number of selected data events and divides by the square root of the selected template count. Please state explicitly whether R is a fixed reference sample or is resampled in the pseudo-experiments, and whether the finite size of R is accounted for in the denominator.
- [Throughout] There are several typographical and grammatical issues, including 'a end-to-end,' 'subjettinness,' 'F ALKON' with an internal space, and 'hyperparameters choice.' A careful proofreading pass is needed.
- [§3.1 and §5] The paper mentions the look-elsewhere effect when scanning the resonant variable but does not compute a global p-value for the sliding-window scan. Please clarify whether the reported results are local p-values only and whether global p-value combination is intended as future work.
Circularity Check
No circular derivation: the NPLM-vs-BDT comparison is an empirical benchmark on public data; self-citations supply the method, not the conclusion.
full rationale
The paper's central claim (NPLM-based methods outperform BDT-based classifiers at low signal injection) is established by numerical power studies on the public LHCO and RODEM datasets, with calibrations from signal-free pseudo-experiments (Sec. 4.3, Figs. 3-5, Table 1). No step in the paper defines the claimed advantage in terms of its own inputs: Eq. (5) is the standard NPLM test statistic and Eq. (6) is the empirical p-value calibration, both from prior published work; the reported power curves are measured outcomes, not quantities forced by construction. The authors cite their own earlier work for the NPLM algorithm [15,16,17] and for the multiple-testing hyper-test [23], but these citations supply the method being benchmarked rather than the empirical conclusion. The paper explicitly says 'R in this work pertains to the idealised setting' (Sec. 4.2) and acknowledges that end-to-end NPLM requires 'a perfect modelling of the background template' (Sec. 4.4); the fact that the NPLM-classifier's advertised use case (imperfect background modelling) is not stress-tested is a scope/validity limitation, not a circularity. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation that carries the result. Score 2 reflects the presence of numerous self-citations that are method-supplying but not load-bearing for the empirical finding.
Assumptions & free parameters
free parameters (6)
- NPLM Gaussian kernel width sigma =
not reported (multiple testing over five values per [23])
- NPLM number of kernels M =
not reported
- NPLM L2 regularization coefficient lambda =
not reported
- BDT maximum leaf nodes nleaf =
31 or 100
- BDT L2 regularization lambda =
1e-2 or 1e-1
- BDT selection threshold thr =
0.8, 0.9, 0.95, 0.99
assumptions (6)
- standard math Neyman-Pearson lemma provides the optimal test statistic
- domain assumption Perfect background template is available in the signal region
- domain assumption NPLM null distribution calibrated by pseudo-experiments is valid
- domain assumption Multiple testing over sigma from [23] preserves test validity
- domain assumption LHCO and RODEM simulations accurately model detector response
- ad hoc to paper In-sample training is beneficial for rare signal detection
Cite this review
Pith. "Pith review of Robust resonant anomaly detection with NPLM." pith.science (2026). https://pith.science/paper/CYBVLD72
@misc{pith2026250101778,
author = {Pith},
title = {Pith review of: Robust resonant anomaly detection with NPLM},
year = {2026},
howpublished = {\url{https://pith.science/paper/CYBVLD72}},
note = {Machine review of arXiv:2501.01778}
}
read the original abstract
In this study, we investigate the application of the New Physics Learning Machine (NPLM) algorithm as an alternative to the standard CWoLa method with Boosted Decision Trees (BDTs), particularly for scenarios with rare signal events. NPLM offers an end-to-end approach to anomaly detection and hypothesis testing by utilizing an in-sample evaluation of a binary classifier to estimate a log-density ratio, which can improve detection performance without prior assumptions on the signal model. We examine two approaches: (1) a end-to-end NPLM application in cases with reliable background modelling and (2) an NPLM-based classifier used for signal selection when accurate background modelling is unavailable, with subsequent performance enhancement through a hyper-test on multiple values of the selection threshold. Our findings show that NPLM-based methods outperform BDT-based approaches in detection performance, particularly in low signal injection scenarios, while significantly reducing epistemic variance due to hyperparameter choices. This work highlights the potential of NPLM for robust resonant anomaly detection in particle physics, setting a foundation for future methods that enhance sensitivity and consistency under signal variability.
Reference graph
Works this paper leans on
-
[23]
Multiple testing for signal-agnostic searches of new physics with machine learning
Gaia Grosso and Marco Letizia. Multiple testing for signal-agnostic searches of new physics with machine learning. 8 2024, 2408.12296
work page Pith review arXiv 2024
-
[1]
Anomaly Detection with Density Estimation
Benjamin Nachman and David Shih. Anomaly Detection with Density Estimation. Phys. Rev. D, 101:075042, 2020, 2001.04990
arXiv 2020
-
[2]
Classifying anomalies through outer density estimation
Anna Hallin, Joshua Isaacson, Gregor Kasieczka, Claudius Krause, Benjamin Nachman, Tobias Quadfasel, Matthias Schlaffer, David Shih, and Manuel Sommerhalder. Classifying anomalies through outer density estimation. Phys. Rev. D, 106(5):055006, 2022
work page 2022
-
[3]
Res- onant anomaly detection without background sculpting
Anna Hallin, Gregor Kasieczka, Tobias Quadfasel, David Shih, and Manuel Sommerhalder. Res- onant anomaly detection without background sculpting. 10 2022, 2210.14924
arXiv 2022
-
[4]
John Andrew Raine, Samuel Klein, Debajyoti Sengupta, and Tobias Golling. CURTAINs for your sliding window: Constructing unobserved regions by transforming adjacent intervals.Front. Big Data, 6:899345, 2023, 2203.09470
arXiv 2023
-
[5]
CURTAINs Flows For Flows: Constructing Unobserved Regions with Maximum Likelihood Estimation
Debajyoti Sengupta, Samuel Klein, John Andrew Raine, and Tobias Golling. CURTAINs Flows For Flows: Constructing Unobserved Regions with Maximum Likelihood Estimation. 5 2023, 2305.04646
arXiv 2023
-
[6]
Improving new physics searches with diffusion models for event observables and jet constituents
Debajyoti Sengupta, Matthew Leigh, John Raine, Samuel Klein, and Tobias Golling. Improving new physics searches with diffusion models for event observables and jet constituents. Journal of High Energy Physics, 2024, 04 2024
work page 2024
-
[7]
Residual anode, 2023, 2312.11629
Ranit Das, Gregor Kasieczka, and David Shih. Residual anode, 2023, 2312.11629
arXiv 2023
Show all 34 references
-
[8]
Complete optimal non-resonant anomaly detection, 2024, 2404.07258
Gregor Kasieczka, John Andrew Raine, David Shih, and Aman Upadhyay. Complete optimal non-resonant anomaly detection, 2024, 2404.07258
2024 arXiv
-
[9]
Model-agnostic search for dijet resonances with anomalous jet substructure in proton-proton collisions at √s = 13 TeV. 2024
2024
-
[10]
Dijet resonance search with weak supervision using √s = 13 TeV pp collisions in the ATLAS detector
ATLAS Collaboration. Dijet resonance search with weak supervision using √s = 13 TeV pp collisions in the ATLAS detector. Phys. Rev. Lett., 125(13):131801, 2020, 2005.02983
2020 arXiv
-
[11]
Goodness of fit by neyman- pearson testing
Gaia Grosso, Marco Letizia, Maurizio Pierini, and Andrea Wulzer. Goodness of fit by neyman- pearson testing. SciPost Physics, 16(5), May 2024
2024
-
[12]
Metodiev, Benjamin Nachman, and Jesse Thaler
Eric M. Metodiev, Benjamin Nachman, and Jesse Thaler. Classification without labels: Learning from mixed samples in high energy physics. JHEP, 10:174, 2017, 1708.02949
2017 arXiv
-
[13]
Collins, Kiel Howe, and Benjamin Nachman
Jack H. Collins, Kiel Howe, and Benjamin Nachman. Extending the search for new resonances with machine learning. Phys. Rev. D, 99(1):014038, 2019, 1902.02634. 14 Robust resonant anomaly detection with NPLM
2019 arXiv
-
[14]
Back To The Roots: Tree-Based Algorithms for Weakly Supervised Anomaly Detection
Thorben Finke, Marie Hein, Gregor Kasieczka, Michael Kr ¨amer, Alexander M ¨uck, Parada Prangchaikul, Tobias Quadfasel, David Shih, and Manuel Sommerhalder. Back To The Roots: Tree-Based Algorithms for Weakly Supervised Anomaly Detection. 9 2023, 2309.13111
2023 arXiv
-
[15]
Learning New Physics from a Machine
Raffaele Tito D’Agnolo and Andrea Wulzer. Learning New Physics from a Machine. Phys. Rev. D, 99(1):015014, 2019, 1806.02350
2019 arXiv
-
[16]
Learning multivariate new physics
Raffaele Tito D’Agnolo, Gaia Grosso, Maurizio Pierini, Andrea Wulzer, and Marco Zanetti. Learning multivariate new physics. Eur. Phys. J. C, 81(1):89, 2021, 1912.12155
2021 arXiv
-
[17]
Learning new physics efficiently with nonparametric methods
Marco Letizia, Gianvito Losapio, Marco Rando, Gaia Grosso, Andrea Wulzer, Maurizio Pierini, Marco Zanetti, and Lorenzo Rosasco. Learning new physics efficiently with nonparametric methods. Eur. Phys. J. C, 82(10):879, 2022, 2204.02317
2022 arXiv
-
[18]
Jerzy Neyman, Egon Sharpe Pearson, and Karl Pearson. Ix. on the problem of the most ef- ficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character , 231(694-706):289–33...
1933
-
[19]
Learning new physics from an imperfect machine
Raffaele Tito d’Agnolo, Gaia Grosso, Maurizio Pierini, Andrea Wulzer, and Marco Zanetti. Learning new physics from an imperfect machine. Eur. Phys. J. C, 82(3):275, 2022, 2111.13633
2022 arXiv
-
[20]
Falkon: An optimal large scale kernel method
Alessandro Rudi, Luigi Carratino, and Lorenzo Rosasco. Falkon: An optimal large scale kernel method. Advances in neural information processing systems, 30, 2017
2017
-
[21]
Kernel methods through the roof: Handling billions of points efficiently
Giacomo Meanti, Luigi Carratino, Lorenzo Rosasco, and Alessandro Rudi. Kernel methods through the roof: Handling billions of points efficiently. In H. Larochelle, M. Ranzato, R. Had- sell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , vo...
2020
-
[22]
Searching for unexpected New Physics at the LHC with Machine Learning
Gaia Grosso. Searching for unexpected New Physics at the LHC with Machine Learning . PhD thesis, U. Padua (main), Padua U., 1 2023
2023
-
[24]
Official Datasets for LHC Olympics 2020 Anomaly Detection Challenge (Version v6), 2019
Gregor Kasieczka, Benjamin Nachman, and David Shih. Official Datasets for LHC Olympics 2020 Anomaly Detection Challenge (Version v6), 2019
2020
-
[25]
Christiansen, Richard Corke, Nishita Desai, Philip Il- ten, Stephen Mrenna, Stefan Prestel, Christine O
Torbj ¨orn Sj ¨ostrand, Stefan Ask, Jesper R. Christiansen, Richard Corke, Nishita Desai, Philip Il- ten, Stephen Mrenna, Stefan Prestel, Christine O. Rasmussen, and Peter Z. Skands. An intro- duction to pythia 8.2. Computer Physics Communications, 191:159–177, June 2015
2015
-
[26]
de Favereau, C
J. de Favereau, C. Delaere, P . Demin, A. Giammanco, V . Lemaˆıtre, A. Mertens, and M. Selvaggi. DELPHES 3, A modular framework for fast simulation of a generic collider experiment. JHEP, 02:057, 2014, 1307.6346
2014 arXiv
-
[27]
The anti-ktjet clustering algorithm
Matteo Cacciari, Gavin P Salam, and Gregory Soyez. The anti-ktjet clustering algorithm. Journal of High Energy Physics, 2008(04):063–063, April 2008. 15 Robust resonant anomaly detection with NPLM
2008
-
[28]
Salam, and Gregory Soyez
Matteo Cacciari, Gavin P . Salam, and Gregory Soyez. FastJet User Manual. Eur. Phys. J. C , 72:1896, 2012, 1111.6097
2012 arXiv
-
[29]
Rodem jet datasets, 2024, 2408.11616
Knut Zoch, John Andrew Raine, Debajyoti Sengupta, and Tobias Golling. Rodem jet datasets, 2024, 2408.11616
2024 arXiv
-
[30]
Christensen, C ´eline Degrande, Claude Duhr, and Benjamin Fuks
Adam Alloul, Neil D. Christensen, C ´eline Degrande, Claude Duhr, and Benjamin Fuks. Feyn- rules 2.0 — a complete toolbox for tree-level phenomenology.Computer Physics Communications, 185(8):2250–2300, August 2014
2014
-
[31]
Identifying boosted objects with n-subjettiness
Jesse Thaler and Ken Van Tilburg. Identifying boosted objects with n-subjettiness. Journal of High Energy Physics, 2011(3), March 2011
2011
-
[32]
Pedregosa, G
F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P . Pret- tenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python.Journal of Machine Lear...
2011
-
[33]
Kernel methods through the roof: handling billions of points efficiently
Giacomo Meanti, Luigi Carratino, Lorenzo Rosasco, and Alessandro Rudi. Kernel methods through the roof: handling billions of points efficiently. In Advances in Neural Information Pro- cessing Systems 32, 2020
2020
-
[34]
Efficient hyperpa- rameter tuning for large scale kernel ridge regression
Giacomo Meanti, Luigi Carratino, Ernesto De Vito, and Lorenzo Rosasco. Efficient hyperpa- rameter tuning for large scale kernel ridge regression. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, 2022. 16
2022
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.