REVIEW 3 major objections 4 minor 1 cited by
Communicating Likelihoods with Normalising Flows
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that a normalising flow trained on unbinned likelihood samples and validated by a joint-distribution test can serve as a compact, reliable surrogate likelihood for reuse in downstream physics analyses.
desk verdict Useful workflow paper, but the headline validation test checks only the radial marginal, so the reliability claim is overstated; fixable before publication. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the invertible normalising flow $f: T \to B$ together with the base-space standard normal distribution. Training minimises the negative log-density of the base-space standard normal evaluated on transformed training samples, which forces the learned density in target space to approximate the empirical distribution of the samples. Validation then checks the claim from the other direction: if the flow has learned the joint distribution, the transformed test samples must look standard normal, and the squared two-norm of a $D$-dimensional standard normal follows a $\chi^2$ distribution with $D$ degrees of freedom. The paper uses masked autoregressive flows with rational-quadratic-spline bijections as its default architecture and stores the trained flow plus any coordinate preprocessing in a single file.
What would settle it
Train a normalising flow on samples drawn from a multivariate normal with strong correlations, but withhold the correlation structure during training, for example by training on the same marginals reshuffled independently. The workflow's KS test on $||\vec{\beta}||^2$ would still report a large p-value because the radial distribution matches a standard normal, even though the joint distribution is plainly wrong. A direct check would be to compare the flow's implied correlations (or a likelihood-ratio test on held-out data) against the true correlations; if such a mismatch survives while the KS p-value stays above threshold, the validation test is not sufficient.
Extended reading notes
Core claim
The central claim is that a normalising flow trained on samples can serve as a reliable proxy for the true unbinned likelihood, provided it passes a rigorous joint-distribution test after training. In the proposed training, the flow $f$ maps target-space samples $\vec{\vartheta}$ to base-space samples $\vec{\beta}$; if the model is perfect, the $\vec{\beta}$ are $D$-dimensional standard normal. The paper's primary validation is the Kolmogorov-Smirnov test applied to the squared two-norm $||\vec{\beta}||^2$, whose distribution is $\chi^2_D$ under the null hypothesis, so the test has a known CDF and no Monte-Carlo uncertainty. Passing this test at a preset threshold (default 3%) is the criterion that qualifies the model as a likelihood model. The authors report acceptable p-values in all three examples and argue this makes the stored flow a faithful, reusable representation of the original likelihood.
Load-bearing premise
The workflow's reliability rests on the assumption that the Kolmogorov-Smirnov test of $||\vec{\beta}||^2$ can detect any meaningful mismatch between the learned and true joint distribution; however, that test only compares radial behaviour, so a flow that gets the correlations between variables wrong could still pass it.
Editorial extensions
If this is right
- Experimental and phenomenological likelihoods can be preserved as small single-file models: the paper reports 4MB for the ATLAS example versus 14MB for compressed samples, and 90kB versus 5MB for the WET example.
- A validated likelihood model can be used directly for density evaluation and sample generation, so downstream analyses can proceed without the original data or analysis code.
- The five-dimensional WET posterior becomes usable in BSM and SMEFT studies without repeating the expensive fit that included 50 hadronic nuisance parameters.
- The per-epoch KS test provides a concrete pass/fail stop for training, removing Monte-Carlo uncertainty from the validation step because the null CDF is known analytically.
Reading between the lines
- A stronger validation would also test directional structure, for example by applying the KS idea to random one-dimensional projections of the base-space samples or to binned full-dimensional statistics; the present radial test alone may not catch misspecified correlations.
- The same train-and-validate pattern could be applied to surrogate models of simulation workflows, so that a validated flow replaces not only the likelihood but also parts of the Monte-Carlo pipeline in iterative inference.
- One could benchmark the workflow by intentionally training on a misspecified model and checking whether the KS test's p-value actually drops when the angular structure is wrong; that experiment would quantify the test's sensitivity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a workflow for building 'likelihood models' from pre-existing samples using normalizing flows. The authors train an invertible map that sends samples into a standard-normal base space, evaluate the KL-equivalent loss, and validate the learned density by testing whether the squared two-norm of the base-space samples follows a chi-square distribution, using both an unbinned Kolmogorov-Smirnov test and a binned chi-square test. They demonstrate the workflow on three high-energy-physics examples: ATLAS Z+jets data, an LHCb Dalitz-plot analysis, and posterior samples of weak-effective-theory parameters from a flavour fit. They also introduce an open-source implementation, nabu. The central claims are that the workflow enables reliable communication of likelihoods and that the validation step constitutes 'rigorous statistical tests of the joint distribution.'
Significance. If the central claims were fully supported, the workflow would be a practically valuable contribution to analysis preservation and reinterpretation: likelihoods could be distributed as compact flow models, evaluated without the original analysis code, and used in downstream inference. The paper has concrete strengths: it ships an open-source implementation, demonstrates the idea on three real examples of different dimensionality, reports storage-size comparisons, and the three examples appear internally consistent with the stated training procedure. These strengths make the paper worth pursuing, but the current validation and the likelihood-versus-posterior distinction are not yet at the level that would justify the reliability claims in the abstract and conclusion.
major comments (3)
- [Section II, 'Testing'] The primary validation test is not a joint-distribution test, contrary to the claim that the KS test on the squared two-norm 'tests the joint distribution.' For a D-dimensional standard normal, ||beta||^2 is the radial marginal; any density of the form p(beta) = chi2_D(||beta||^2) * q(beta/||beta||) / S_{D-1} with arbitrary angular distribution q produces exactly the same chi-square distribution of ||beta||^2. A flow that has learned the correct radial profile but wrong angular correlations would therefore pass the reported KS and binned chi-square tests while defining an incorrect joint density. Since the abstract and Section II rest the reliability claim on this test, the validation needs either a genuinely joint test (for example, a classifier two-sample test, energy distance, maximum mean discrepancy, or multiple random projections) or a substantially weakened claim about what is being validated.
- [Section III, example (c)] The workflow in example (c) trains on posterior samples of weak-effective-theory parameters obtained from a Bayesian fit, and the paper then states that these results can be used in subsequent BSM or SMEFT studies without repeating the original analysis. A density learned from posterior samples is the posterior density, not the likelihood: it includes the prior and the marginalisation over the 50 hadronic nuisance parameters. Substituting this density for a likelihood in a new fit would double-count the prior and would not reproduce the original likelihood's dependence on the parameters. The manuscript needs to either clarify the precise inferential use of the learned model in this example or replace it with an example where the samples are genuine likelihood samples (for example, Monte Carlo events drawn at fixed parameter values).
- [General workflow, Section II and Introduction] The manuscript does not specify what 'likelihood' means when the input is a single unlabelled sample set. Equation (2) trains a density estimator on the input samples, so the learned object is the density of those samples, p(vartheta). For a likelihood usable in parameter inference, one needs the probability of the data as a function of physics parameters, which generally requires samples at multiple parameter points or a conditional flow. The examples are ambiguous: in (a) the input appears to be measured events, in (c) it is posterior parameter samples. The authors should state explicitly what inferential operations the learned LM supports and, if the claim is only that it approximates the density of a fixed data set, say so directly.
minor comments (4)
- [Section II, 'Testing'] There is a typo: 'test static' should read 'test statistic.'
- [Section II, 'Testing'] The notation '||vec(beta)_i||^2_2' is redundant; the squared two-norm is already ||vec(beta)_i||^2.
- [Section II, 'Testing'] The statement that the binned chi-square test uses the normal approximation for bin counts could be clarified by noting that the approximation requires the expected counts to be sufficiently large, as the authors do state with N/K >= 100.
- [Discussion around Fig. 1] The p-values reported for each example are for the test on the testing set only; it would be helpful to state explicitly whether the same conclusion holds for the validation set, since early stopping is based on the validation loss.
Circularity Check
No significant circularity: the workflow trains a normalizing flow on samples and validates on held-out data; self-citations are not load-bearing.
full rationale
The claimed derivation is an empirical fitting procedure, not a derivation of a result from its own inputs. The normalizing flow is trained by minimizing the negative log standard-normal likelihood in base space (Eq. 2) and then checked on held-out samples with a KS test on the squared two-norm and a binned chi-square test. The held-out testing set makes the validation a genuine out-of-sample check; no fitted parameter is relabeled as a prediction. The only self-references are the pilot study Ref. [29] and the authors' own code nabu [46]; Ref. [29] is cited for terminology and as a baseline to contrast with, and the present paper reports new p-values on ATLAS, LHCb, and WET data, so the central claim does not reduce to the self-citation. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. I flag one non-circular limitation: Section II claims the KS test on the squared two-norm 'tests the joint distribution,' but the squared two-norm is a one-dimensional radial marginal; a flow with wrong angular correlations could pass the test. This is a statistical validity concern about the strength of the validation, not a circularity, and does not affect the circularity score.
Assumptions & free parameters
free parameters (3)
- KS rejection threshold =
3%
- Z+jets observable selection =
p_mu_mu_T, y_mu_mu, p_mu1_T, Δη²₁₂, Δϕ²₁₂
- Flow architecture hyperparameters =
8 layers, 512 hidden units, 12 RQS knots, learning rate 1% with decay, early stopping patience 50
assumptions (4)
- standard math After successful training, base-space samples follow a standard normal distribution, and the squared two-norm follows χ²_D.
- domain assumption Training samples are representative draws from the likelihood being communicated.
- ad hoc to paper A density learned from posterior samples can serve as a likelihood in subsequent analyses.
- domain assumption The five chosen observables capture all physical information in the 24-dimensional ATLAS dataset.
Cite this review
Pith. "Pith review of Communicating Likelihoods with Normalising Flows." pith.science (2026). https://pith.science/paper/F4DR4LZY
@misc{pith2026250209494,
author = {Pith},
title = {Pith review of: Communicating Likelihoods with Normalising Flows},
year = {2026},
howpublished = {\url{https://pith.science/paper/F4DR4LZY}},
note = {Machine review of arXiv:2502.09494}
}
read the original abstract
We present a machine-learning-based workflow to model an unbinned likelihood from its samples. A key advancement over existing approaches is the validation of the learned likelihood using rigorous statistical tests of the joint distribution, such as the Kolmogorov-Smirnov test of the joint distribution. Our method enables the reliable communication of experimental and phenomenological likelihoods for subsequent analyses. We demonstrate its effectiveness through three case studies in high-energy physics. To support broader adoption, we provide an open-source reference implementation, nabu.
Figures
Forward citations
Cited by 1 Pith paper
-
Toward an event-level analysis of hadron structure using differential programming
LOITS is a differentiable sampling method, demonstrated in a GAN closure test, that maps sampled events back to the parameters of a target density for event-level inference.
Reference graph
Works this paper leans on
-
[1]
G. Alguero, J. Y. Araz, B. Fuks, and S. Kraml, SciPost Phys. 14, 009 (2023), arXiv:2206.14870 [hep-ph]
arXiv 2023
-
[2]
J. Y. Araz, B. Fuks, M. D. Goodsell, and M. Utsch, Eur. Phys. J. C 82, 597 (2022), arXiv:2112.05163 [hep-ph]
arXiv 2022
-
[3]
J. Y. Araz, B. Fuks, and G. Polykratis, Eur. Phys. J. C 81, 329 (2021), arXiv:2006.09387 [hep-ph]
arXiv 2021
-
[4]
E. Conte and B. Fuks, Int. J. Mod. Phys. A 33, 1830027 (2018), arXiv:1808.00480 [hep-ph]
arXiv 2018
- [5]
- [6]
- [7]
-
[8]
J. Y. Araz, A. Buckley, and B. Fuks, Eur. Phys. J. C 83, 664 (2023), arXiv:2303.03427 [hep-ph]
arXiv 2023
Show all 62 references
-
[9]
Bierlich et al
C. Bierlich et al. , SciPost Phys. 8, 026 (2020), arXiv:1912.05451 [hep-ph]
2020 arXiv
-
[10]
Buckley, D
A. Buckley, D. Kar, and K. Nordstr¨ om, SciPost Phys. 8, 025 (2020), arXiv:1910.01637 [hep-ph]
2020 arXiv
-
[11]
Buckley, J
A. Buckley, J. Butterworth, D. Grellscheid, H. Hoeth, L. Lonnblad, J. Monk, H. Schulz, and F. Siegert, Com- put. Phys. Commun. 184, 2803 (2013), arXiv:1003.0694 [hep-ph]
2013 arXiv
-
[12]
Alguero, J
G. Alguero, J. Heisig, C. K. Khosa, S. Kraml, S. Kulka- rni, A. Lessa, H. Reyes-Gonz´ alez, W. Waltenberger, and A. Wongel, JHEP 08, 068 (2022), arXiv:2112.00769 [hep- ph]
2022 arXiv
-
[13]
Alguero, S
G. Alguero, S. Kraml, and W. Waltenberger, Comput. Phys. Commun. 264, 107909 (2021), arXiv:2009.01809 [hep-ph]
2021 arXiv
-
[14]
Ambrogi et al., Comput
F. Ambrogi et al., Comput. Phys. Commun. 251, 106848 (2020), arXiv:1811.10624 [hep-ph]
2020 arXiv
-
[15]
Cern open data project,
CERN, “Cern open data project,” https://opendata. cern.ch/ (2014–2024)
2014
-
[16]
C. A. Aidala, C. Burr, M. Cattaneo, D. S. Fitzgerald, A. Morris, S. Neubert, and D. Tropmann, Comput. Softw. Big Sci. 7, 6 (2023), arXiv:2302.14235 [hep-ex]
2023 arXiv
-
[17]
Heinrich, M
L. Heinrich, M. Feickert, G. Stark, and K. Cranmer, J. Open Source Softw. 6, 2823 (2021)
2021
-
[18]
Cranmer, G
K. Cranmer, G. Lewis, L. Moneta, A. Shibata, and W. Verkerke (ROOT), (2012)
2012
-
[19]
J. Y. Araz, SciPost Phys. 16, 032 (2024), arXiv:2307.06996 [hep-ph]
2024 arXiv
-
[20]
Flacher, M
H. Flacher, M. Goebel, J. Haller, A. Hocker, K. Monig, and J. Stelzer, Eur. Phys. J. C 60, 543 (2009), [Erratum: Eur.Phys.J.C 71, 1718 (2011)], arXiv:0811.0009 [hep-ph]
2009 arXiv
-
[21]
Englert, R
C. Englert, R. Kogler, H. Schulz, and M. Spannowsky, Eur. Phys. J. C 76, 393 (2016), arXiv:1511.05170 [hep- ph]
2016 arXiv
-
[22]
Englert, R
C. Englert, R. Kogler, H. Schulz, and M. Spannowsky, Eur. Phys. J. C 77, 789 (2017), arXiv:1708.06355 [hep- ph]
2017 arXiv
-
[23]
Drees, H
M. Drees, H. Dreiner, D. Schmeier, J. Tattersall, and J. S. Kim, Comput. Phys. Commun. 187, 227 (2015), arXiv:1312.2591 [hep-ph]
2015 arXiv
-
[24]
Dercks, N
D. Dercks, N. Desai, J. S. Kim, K. Rolbiecki, J. Tatter- sall, and T. Weber, Comput. Phys. Commun. 221, 383 (2017), arXiv:1611.09856 [hep-ph]
2017 arXiv
-
[25]
Athron et al
P. Athron et al. (GAMBIT), Eur. Phys. J. C 77, 784 (2017), [Addendum: Eur.Phys.J.C 78, 98 (2018)], arXiv:1705.07908 [hep-ph]
2017 arXiv
-
[26]
Amrith, J
S. Amrith, J. M. Butterworth, F. F. Deppisch, W. Liu, A. Varma, and D. Yallup, JHEP 05, 154 (2019), arXiv:1811.11452 [hep-ph]
2019 arXiv
-
[27]
Coccaro, M
A. Coccaro, M. Pierini, L. Silvestrini, and R. Torre, Eur. Phys. J. C 80, 664 (2020), arXiv:1911.03305 [hep-ph]
2020 arXiv
-
[28]
Reyes-Gonzalez and R
H. Reyes-Gonzalez and R. Torre, SciPost Phys. Core 7, 048 (2024), arXiv:2309.09743 [hep-ph]
2024 arXiv
-
[29]
A. Beck, M. Reboud, and D. van Dyk, Eur. Phys. J. C 83, 1115 (2023), arXiv:2309.10365 [hep-ph]
2023 arXiv
-
[30]
Heinrich, (2022), arXiv:2203.13079 [stat.ME]
L. Heinrich, (2022), arXiv:2203.13079 [stat.ME]
2022 arXiv
-
[31]
R. E. C. Smith, I. Ochoa, R. In´ acio, J. Shoemaker, and M. Kagan, Phys. Rev. D 110, 052010 (2024), arXiv:2310.12804 [hep-ex]
2024 arXiv
- [32]
-
[33]
Heimel, R
T. Heimel, R. Winterhalder, A. Butter, J. Isaacson, C. Krause, F. Maltoni, O. Mattelaer, and T. Plehn, Sci- Post Phys. 15, 141 (2023), arXiv:2212.06172 [hep-ph]
2023 arXiv
- [34]
-
[35]
Kofler, V
A. Kofler, V. Stimper, M. Mikhasenko, M. Kagan, and L. Heinrich, in 38th conference on Neural Information Processing Systems (2024) arXiv:2411.16234 [hep-ph]
2024 arXiv
-
[36]
M. Vigl, N. Hartman, and L. Heinrich, Mach. Learn. Sci. Tech. 5, 025075 (2024), arXiv:2401.13536 [hep-ex]
2024 arXiv
-
[37]
Heimel, N
T. Heimel, N. Huetsch, F. Maltoni, O. Mattelaer, T. Plehn, and R. Winterhalder, SciPost Phys. 17, 023 (2024), arXiv:2311.01548 [hep-ph]
2024 arXiv
-
[38]
Heimel, O
T. Heimel, O. Mattelaer, T. Plehn, and R. Winterhalder, SciPost Phys. 18, 017 (2025), arXiv:2408.01486 [hep-ph]
2025 arXiv
-
[39]
Cranmer, J
K. Cranmer, J. Brehmer, and G. Louppe, Proc. Nat. Acad. Sci. 117, 30055 (2020), arXiv:1911.01429 [stat.ML]
2020 arXiv
-
[40]
Brehmer, F
J. Brehmer, F. Kling, I. Espejo, and K. Cranmer, Com- put. Softw. Big Sci. 4, 3 (2020), arXiv:1907.10621 [hep- ph]
2020 arXiv
-
[41]
Brehmer, K
J. Brehmer, K. Cranmer, G. Louppe, and J. Pavez, Phys. Rev. D 98, 052004 (2018), arXiv:1805.00020 [hep-ph]
2018 arXiv
-
[42]
Brehmer, K
J. Brehmer, K. Cranmer, G. Louppe, and J. Pavez, Phys. Rev. Lett. 121, 111801 (2018), arXiv:1805.00013 [hep- ph]
2018 arXiv
-
[43]
Mastandrea, B
R. Mastandrea, B. Nachman, and T. Plehn, Phys. Rev. D 110, 056004 (2024), arXiv:2405.15847 [hep-ph]
2024 arXiv
-
[44]
M. Dax, J. Wildberger, S. Buchholz, S. R. Green, J. H. Macke, and B. Sch¨ olkopf, (2023), arXiv:2305.17161 [cs.LG]
2023 arXiv
-
[45]
J. Y. Araz, A. Buckley, G. Kasieczka, J. Kieseler, S. Kraml, A. Kvellestad, A. Lessa, T. Procter, A. Rak- lev, H. Reyes-Gonzalez, K. Rolbiecki, S. Sekmen, and G. Unel, SciPost Phys. Comm. Rep. , 3 (2024)
2024
-
[46]
J. Y. Araz, D. van Dyk, and . . . , “Nabu,”
-
[47]
Flowjax: Distributions and normalizing flows in jax,
D. Ward, “Flowjax: Distributions and normalizing flows in jax,” ([release year of version])
-
[48]
Kidger and C
P. Kidger and C. Garcia, Differentiable Programming workshop at Neural Information Processing Systems 2021 (2021)
2021
-
[49]
JAX: com- posable transformations of Python+NumPy programs,
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. Vander- 6 Plas, S. Wanderman-Milne, and Q. Zhang, “JAX: com- posable transformations of Python+NumPy programs,” (2018)
2018
-
[50]
Papamakarios, T
G. Papamakarios, T. Pavlakou, and I. Murray, in Advances in Neural Information Processing Systems , Vol. 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Inc., 2017)
2017
-
[51]
L. Dinh, J. Sohl-Dickstein, and S. Bengio, in Interna- tional Conference on Learning Representations (2017)
2017
-
[52]
Neural spline flows,
C. Durkan, A. Bekasov, I. Murray, and G. Papamakar- ios, “Neural spline flows,” (2019), arXiv:1906.04032 [stat.ML]
2019 arXiv
-
[53]
D. P. Kingma and J. Ba (2014) arXiv:1412.6980 [cs.LG]
2014 arXiv
-
[54]
Kolmogorov–smirnov test,
“Kolmogorov–smirnov test,” in The Concise Encyclope- dia of Statistics (Springer New York, New York, NY,
-
[55]
Atlas omnifold 24-dimensional z+jets open data,
A. Collaboration, “Atlas omnifold 24-dimensional z+jets open data,” (2024)
2024
-
[56]
Aad et al
G. Aad et al. (ATLAS), Phys. Rev. Lett. 133, 261803 (2024), arXiv:2405.20041 [hep-ex]
2024 arXiv
-
[57]
Aaij et al
R. Aaij et al. (LHCb), Phys. Rev. D 102, 112003 (2020), arXiv:2009.00026 [hep-ex]
2020
-
[58]
EOS/DATA-2023-01: Supplementary material for EOS/ANALYSIS-2022-05,
D. Leljak, B. Meli´ c, F. Novak, M. Reboud, and D. van Dyk, “EOS/DATA-2023-01: Supplementary material for EOS/ANALYSIS-2022-05,” (2023)
2023
-
[59]
Leljak, B
D. Leljak, B. Meli´ c, F. Novak, M. Reboud, and D. van Dyk, JHEP 08, 063 (2023), arXiv:2302.05268 [hep-ph]
2023 arXiv
-
[60]
van Dyk et al
D. van Dyk et al. (EOS Authors), Eur. Phys. J. C 82, 569 (2022), arXiv:2111.15428 [hep-ph]
2022 arXiv
-
[61]
Bolognani, M
C. Bolognani, M. Reboud, D. van Dyk, and K. K. Vos, JHEP 09, 099 (2024), arXiv:2407.06145 [hep-ph]
2024 arXiv
- [62]
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.