{"id":"9f067ae8-1683-4b3d-ae26-947f9ba7f577","arxiv_id":"2505.08709","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hybrid of contrastive normalizing flows and a deep classifier yields signal-strength estimates with narrower confidence intervals than a likelihood-based method on the HiggsML Uncertainty Challenge dataset.","lead":"This paper combines contrastive normalizing flows, which learn signal-focused probability densities, with a deep classifier to estimate the Higgs signal fraction while accounting for detector systematics, and it reports calibrated confidence intervals on the HiggsML Uncertainty Challenge dataset. It matters because it offers a practical recipe for uncertainty-aware parameter estimation under domain shifts, a core issue in particle physics and applied machine learning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is conditional on an unshown ablation: at the small nuisance shifts of HiggsML, the toy itself shows a plain DNN beating the CNF-feature classifier, so the reported 0.823 score is not yet attributable to the CNF features.","rationale":"I read the paper as claiming that CNF-derived features provide a material robustness advantage for parameter estimation under nuisance shifts, and that this advantage is what drives the reported HiggsML result. The analytic minimization in Sec. 4.1 is sound and the toy demonstration is a reasonable first check, but the evidence has a regime gap. The toy's small-nuisance panels show the plain DNN outperforming the CNF-feature DNN, yet the challenge's nuisance widths are small. The comparison in Table 2 is therefore not sufficient: the baseline differs in more than the feature set, so it cannot establish feature attribution. This is not an accusation of misreporting; the reported numbers are internally plausible and the Neyman construction is standard. The concern is that the specific load-bearing ingredient, the CNF features, is untested in the regime where the headline claim lives. A clean ablation with and without Phi_s,b, on the same training data and inference procedure, would settle whether the CNF features are central. Without that, the conclusion should be conditional rather than definitive. The reader's weakest assumption identified essentially the same issue, and I agree with that assessment; the additional secondary concerns about missing code and limited external comparison reinforce the need for a checkable ablation. I therefore recommend keeping the verdict CONDITIONAL.","tokens_in":14763,"tokens_out":4321,"duration_ms":48075,"concrete_test":"Run the complete Sec. 6.2 pipeline twice, holding everything fixed (same 1000 perturbed training sub-datasets, same two-headed DNN architecture and training schedule, same Neyman construction and scoring), with and without the Phi_s,b features appended to the 28 kinematic features. Compare score, average 1-sigma interval, coverage, and RMSE on the same 10x100 pseudo-experiments. If the no-Phi run achieves a score within 0.1 of 0.823, or matches interval/coverage within statistical error, the CNF features are not load-bearing. A second complementary check: evaluate both trained DNNs on test sets drawn from a single fixed small nuisance shift representative of Table 3 (e.g., alpha_tes=1.02, alpha_jes=1.02, alpha_soft_met=1.0, other scales at their mean) and compare classification accuracy, to test directly whether the Phi features help in the small-shift regime where Fig.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that CNF-constructed features Phi_s,b make parameter estimation robust to data-distribution distortions. The theoretical derivation (Eq. 10) establishes what a CNF would learn on unperturbed data, but it does not establish invariance of those learned densities under nuisance shifts. The only controlled test of that invariance is the 10D toy in Sec. 5, whose Fig. 3 (center and, by the text, all scenarios) shows the plain DNN (i) is the best classifier at small values of the nuisance parameter, while the DNN with CNF features (iv) only wins at large deformations. The HiggsML challenge operates precisely in the small-shift regime: Table 3 lists prior widths of 0.001 to 0.25 and clipping ranges no wider than a few tens of percent. Thus the mechanism advertised as load-bearing is supported only outside the regime where the headline experiment operates. The reported HiggsML comparison does not resolve this: the 'baseline MLE for mu only' differs from CNF+DNN in both feature set and estimation procedure (it does not profile the nuisance parameters and has much worse coverage), so it is not an ablation that isolates the CNF features. Since the DNN in the proposed method also receives the 28 raw kinematic features, its strong score could plausibly come from learning the small shifts from raw features, with the CNF features adding little. This is the single load-bearing weakness; it is addressable by a clean ablation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a parameter-estimation pipeline for signal-fraction inference in the presence of nuisance parameters. A contrastive normalizing flow (CNF), trained with a loss that penalizes background probability under the signal flow, is used to define discrimination functions Phi_s and Phi_b; these are appended to 28 raw kinematic features and fed to a two-headed DNN classifier. A binned likelihood and Neyman construction are then used to produce coverage-calibrated confidence intervals for the signal strength mu. The authors provide an analytic characterization of the CNF optimum (Eq. 10), validate the approach on a 10D Gaussian toy (Sec. 5), and report results on the HiggsML Uncertainty Challenge (Sec. 6.3, Table 2), claiming a score of 0.823, an average 1-sigma interval of 0.438, coverage 0.672, and roughly 10x faster runtime than a likelihood-based method. The central claim is that CNF features make classification robust to distribution distortions induced by systematic uncertainties.","tokens_in":15057,"tokens_out":5070,"duration_ms":52828,"significance":"If the central claim is established, the paper offers a practical recipe for uncertainty-aware parameter estimation with classifiers in the presence of nuisance parameters, an important problem in HEP and related fields. The contrastive-loss modification is simple and the analytic minimization in Eq. (10) is clean and instructive. The toy study is clearly described, and the HiggsML results are concrete and falsifiable (score, interval length, coverage, runtime). The method's significance, however, rests on whether the CNF-derived features are actually responsible for the reported robustness; currently that attribution is not demonstrated, and the paper's own toy evidence points the other way in the small-shift regime that characterizes the HiggsML benchmark. If the missing isolation is supplied, the results would be a useful contribution; without it, the headline claim is not supported.","major_comments":[{"comment":"The central claim that CNF features drive the benchmark performance is not tested by an ablation. In the only controlled comparison, the plain DNN (i) is the best classifier at small values of the nuisance parameter (Fig. 3, center), while the DNN with CNF features (iv) surpasses other classifiers only for large deformations (Fig. 3, left and right). The HiggsML nuisance priors in Table 3 have widths from 0.001 to 0.25, i.e., the small-shift regime, and the two baselines in Table 2 differ from the proposed method in both feature set and estimation procedure (the 'Baseline MLE for mu only' does not profile nuisances and reports coverage 0.431). Consequently, the 0.823 score cannot currently be attributed to the CNF features. I ask for an ablation that keeps the entire pipeline (raw 28 features, the same DNN training set, the same Neyman construction, and the same profiling of alpha_jes and alpha_tes) fixed and compares performance with and without the CNF-derived Phi_s and Phi_b.","section":"Sec. 5 and Sec. 6.3, Fig. 3, Table 2"},{"comment":"The theoretical argument does not establish robustness under nuisance shifts. Eq. (10) characterizes the density that minimizes the contrastive loss on unperturbed data, but no result is given for how this density, or the associated discrimination functions Phi_s and Phi_b, behaves when ps and pb are deformed by the nuisance parameters. The 'holistic view' argument in Sec. 4.2 is heuristic. Since the paper's central robustness claim is explicitly about data distribution distortions, I ask for either a formal stability statement (e.g., a Lipschitz or bounded-perturbation condition on the learned density as a function of the nuisance parameters) or an empirical demonstration at the small-shift scales actually listed in Table 3.","section":"Sec. 4.2 and Eq. (10)"},{"comment":"The abstract's claim of 'top performance on the HiggsML Uncertainty Challenge' is not supported by an external benchmark comparison. Table 2 compares only three internally constructed methods, and no challenge leaderboard scores, other participant results, or published baselines are reported. Without such a comparison, 'top performance' is unsubstantiated; the paper should either report the method's official challenge rank/score or compare against the publicly available leaderboard and at least one independent baseline using the same scoring function.","section":"Sec. 6.3, Table 2"},{"comment":"Several design choices are made using the target data without a described validation protocol. The removal of 0-jet events, the selection of alpha_jes and alpha_tes as the only profiled nuisance parameters, and the choice of CNF contrast values c=2 and c=0.5 are all justified as 'empirical' in Sec. 6.2. These choices affect the reported score and therefore the claimed robustness. The manuscript should either present a pre-specified or cross-validated selection procedure or show that the headline metrics are stable under reasonable variations of these choices.","section":"Sec. 6.2"}],"minor_comments":[{"comment":"The notation p_s(x | Phi_s[x;{nu_i}]) is not well-defined because Phi_s is itself a function of x; the intended likelihood ratio should be written as a function of the augmented feature vector (x, Phi_s(x), Phi_b(x)), not as a conditional probability on Phi_s.","section":"Sec. 3, Eq. (3)"},{"comment":"The statement that 1000 nuisance-parameter choices correspond to '1000^{1/6} approx 3 choices of each nuisance parameter value' is misleading, since the 1000 sub-datasets are random draws from a 6D distribution, not a grid with three values per parameter. Please rephrase to describe the sampling scheme accurately.","section":"Sec. 3"},{"comment":"There are minor typographical errors: in Eq. (14) the scale function appears with an extra parenthesis, 'exp(s(ub))', and in Appendix E.2 the word 'susbet' should be 'subset'.","section":"Appendix A.1 and Appendix E.2"},{"comment":"The 'Runtime (min)' column is ambiguous as to whether it includes CNF and DNN training time or only the evaluation/Neyman-construction time; please specify exactly what the 5.0 minutes and 41 minutes cover.","section":"Sec. 6.3, Table 2"},{"comment":"The manuscript contains no data or code availability statement. For a benchmark-driven paper, making the training and evaluation pipeline available would substantially improve reproducibility and the usefulness of the comparison.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the pipeline is clearly described, but the headline contribution is not yet isolated: the toy experiment indicates that the advantage of CNF features appears only at large nuisance shifts, whereas the HiggsML benchmark is in the small-shift regime. The missing ablation is fixable and does not require a change of the paper's scope, so I recommend major revision rather than rejection. Please also ask the authors to clarify whether the HiggsML score is an official challenge entry and to provide the external-baseline context for the 'top performance' claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What's new here is not the contrastive loss itself—that comes from Ref. [18]—or the classifier-as-likelihood-ratio idea from Ref. [21]. The contribution is the integration: CNF-derived features fed into a DNN, then a Neyman construction for signal-strength intervals on the HiggsML Uncertainty Challenge. The Table 2 numbers (score 0.823, interval 0.438, coverage 0.672) are credible against their self-built baselines, and the runtime advantage over the likelihood-based method is substantial. The toy experiments in Sec. 5 are clear, and the analytic minimization of the contrastive loss is a useful sanity check. The paper is honest about the toy limitations, which is credit.\n\nThe soft spot is the load-bearing one, and it is largely the authors' own finding. Fig. 3 shows the plain DNN is best at small nuisance values; the CNF-feature DNN only wins at large deformations. The HiggsML nuisance priors in Table 3 have widths 0.001–0.25 with narrow clipping—small shifts. So the headline score is not yet attributable to the CNF features. The two comparison methods differ from CNF+DNN in both feature set and estimation procedure, so they do not isolate the CNF. An ablation with and without the CNF features under an otherwise identical pipeline is the missing experiment. The paper also has no external baselines, no released code, and several hyperparameters (c, k, jet selection, which nuisances to profile) are chosen on the target data without a separate validation set. Those are minor-to-moderate and fixable.\n\nWho this is for: HEP and ML people doing likelihood-free inference or uncertainty-aware classification under domain shift. If I were handling it, I would send it out. The weaknesses are real but not disqualifying; the method is plausible, the benchmark credible, and the central question can be resolved by one clean ablation. Recommendation: accept for peer review, conditional on revision.","headline":"A promising hybrid pipeline whose central claim about CNF robustness is not yet supported in the small-shift regime, but the work is worth refereeing.","tokens_in":15594,"tokens_out":2898,"would_cite":true,"duration_ms":30449,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By encoding signal and background with contrastive normalizing flows, this paper turns learned densities into classifier inputs and reports calibrated frequentist confidence intervals for the Higgs signal strength under nuisance-parameter…","keywords":["contrastive normalizing flows","uncertainty-aware parameter estimation","systematic uncertainties","domain shift","likelihood ratio approximation","frequentist confidence intervals","HiggsML Uncertainty Challenge","signal strength estimation"],"falsifier":"Train the same DNN on the 28 raw features but drop the CNF-derived $\\Phi$ inputs, and evaluate both pipelines on the same 10 by 100 HiggsML pseudo-experiments; if coverage and interval width are statistically indistinguishable, the CNF features are not the load-bearing part of the claimed stability. A second check is to replace the CNF densities with ordinary non-contrastive flow densities and see whether the contrast term, rather than density estimation, is what preserves coverage.","tokens_in":14501,"feed_emoji":"⚛️","tokens_out":8252,"duration_ms":77936,"temperature":0.7,"pith_summary":"Estimating the fraction of signal events in a particle-physics dataset is hard when detector miscalibration deforms the data. This paper proposes a pipeline that first trains contrastive normalizing flows (CNFs) on unperturbed signal and background samples, then feeds their learned densities, along with the raw kinematics, into a neural classifier, and finally converts classifier scores into frequentist confidence intervals via a binned likelihood and confidence-belt construction. The paper shows analytically that the contrastive loss drives the learned density toward a truncated likelihood-ratio contour, and argues this density view is more stable under nuisance-induced shifts than a bare decision boundary. On the HiggsML Uncertainty Challenge, the method reports a score of 0.823, an average 68% confidence interval length of 0.438 at coverage 0.672, roughly half the interval size of a likelihood-based comparison at comparable coverage and about ten times faster. If these numbers hold, the approach offers a practical way to do uncertainty-aware signal-strength fits without modeling a high-dimensional parameter grid.","feed_headline":"CNF pipeline halves Higgs signal error bars and runs 10x faster","feed_subtitle":"On the HiggsML challenge it scores 0.823 with 67% coverage, about half the interval width of likelihood fits.","key_machinery":"The machinery is the contrastive normalizing flow (CNF): a normalizing flow trained with the usual log-likelihood on signal plus a contrastive term $c\\log p_\\theta(x_b)$ on background. In the infinite-data limit the optimum is a truncated likelihood-ratio density, $\\max\\{p_s(x)-c p_b(x),0\\}$ up to normalization, and the boundary where it vanishes is the Bayes-optimal classification contour. The paper builds two such flows per jet category ($c=2$ and $c=0.5$), forms bounded discrimination functions $\\Phi_{s,b}=p_{\\theta}^{(s,b)}/(1+p_{\\theta}^{(s,b)})$, appends them to the 28 kinematic features, and trains a two-headed DNN classifier on 1000 sub-datasets with different nuisance settings. At inference, classifier-score histograms are fit with a binned likelihood in the signal strength $\\mu$ and the two most influential nuisance parameters, and the intervals come from inverting a confidence belt built over many pseudo-experiments.","core_discovery":"The paper claims that the ratio of signal to background densities, approximated through CNF-derived discrimination functions, remains informative under the small nuisance-parameter distortions of the benchmark, so a DNN trained on augmented features can approximate the full likelihood ratio and yield calibrated intervals for the signal strength. The analytic result is that the CNF's learned density minimizes the contrastive loss to $p^{(s)}_{\\theta^*}(x) \\propto \\max\\{p_s(x)-c p_b(x),0\\}$, a truncated likelihood-ratio contour, which the authors interpret as capturing the whole class distribution rather than only a boundary. Empirically, on 10 by 100 pseudo-experiments, CNF+DNN gives score 0.823, mean 68% interval 0.438, coverage 0.672, RMSE 0.191, in 5 minutes, while the likelihood-based method gives score 0.001, interval 0.998, coverage 0.701, in 41 minutes. The paper states the method selects 100 to 3000 signal events out of $10^6$ events with a 1-$\\sigma$ band of 150 to 500 events.","pith_inferences":["Beyond the paper: its own toy study shows the plain DNN is more accurate than the CNF-featured DNN at small nuisance deformations, while the CNF features win at large deformations; since the challenge's nuisance widths are small, this leaves open whether the benchmark gain comes from the CNF densities or from the DNN learning raw-feature shifts.","Beyond the paper: the CNF's generative side suggests a testable extension, namely using the contrastive flow to sample signal-like events that are maximally unlike background and using them as data augmentation, which could improve sensitivity in even rarer-signal regimes.","Beyond the paper: the method's calibration relies on a simulated confidence belt; a natural stress test is to clip or remove the two nuisance-parameter templates and measure how much coverage degrades, which would isolate where the stability claim actually lives.","Beyond the paper: because the pipeline is built on classifier score templates rather than a parametric likelihood, it may transfer to other signal-fraction problems such as rare decays or astroparticle stacks whenever labeled simulations are available, but that transfer is not demonstrated here."],"forward_implications":["If the reported benchmark numbers are correct, signal-strength fits in the HiggsML setting can be made with 68% intervals of length about 0.44, roughly half the width of the likelihood-based comparison at similar coverage.","The 5-minute runtime, versus 41 minutes for the likelihood method, makes large pseudo-experiment coverage studies practical on modest GPU allocations.","Because only two nuisance parameters enter the binned likelihood, the method avoids explicit profiling over all six systematic parameters at test time.","The analytic contrastive optimum ties the hyperparameter $c$ to the truncation contour of the likelihood ratio, giving an interpretable handle for trading signal-richness against stability.","A calibrated pipeline of this kind could be applied to any rare-signal mixture where a simulator can generate labeled signal and background events with nuisance settings."],"supporting_citations":[{"why":"Defines the HiggsML Uncertainty Challenge benchmark, its six nuisance parameters, the signal-strength task, and the score used in the headline comparison.","marker":"[1]"},{"why":"Introduces contrastive normalizing flows and the positive-difference training idea that the paper adapts to signal-fraction estimation.","marker":"[18]"},{"why":"Establishes that calibrated binary classifiers approximate likelihood ratios, the theoretical basis for turning classifier scores into parameter estimates.","marker":"[21]"},{"why":"Supplies the confidence-belt construction used to convert point estimates into coverage-calibrated intervals.","marker":"[22]"},{"why":"Provides the affine coupling normalizing-flow architecture that the CNF training procedure modifies.","marker":"[13]"},{"why":"Documents why naive normalizing-flow likelihoods fail out-of-distribution, motivating the contrastive modification.","marker":"[16]"},{"why":"Uses likelihood ratios for out-of-distribution detection, the conceptual bridge from density estimates to discrimination.","marker":"[17]"}],"fun_headline_variants":["CNFs halve Higgs error bars, run 8x faster","Contrastive flows nail Higgs parameters despite miscalibration","CNF+DNN: tighter Higgs intervals in 5 minutes flat","Uncertainty-aware CNFs beat likelihood fits on HiggsML"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes CNFs trained only on unperturbed signal and background remain discriminative enough when test data are shifted by the benchmark's small nuisance deformations; the paper's own Gaussian toy shows the plain DNN beats the CNF-based classifier precisely in that small-shift regime.","fun_headline_variants_meta":{"raw":{"variants":["CNFs halve Higgs error bars, run 8x faster","Contrastive flows nail Higgs parameters despite miscalibration","CNF+DNN: tighter Higgs intervals in 5 minutes flat","Uncertainty-aware CNFs beat likelihood fits on HiggsML"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1443,"prompt_tokens":995,"completion_tokens":448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":611,"tokens_out":448,"duration_ms":5136,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:48:07.115787+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same DNN on the 28 raw features but drop the CNF-derived $\\Phi$ inputs, and evaluate both pipelines on the same 10 by 100 HiggsML pseudo-experiments; if coverage and interval width are statistically indistinguishable, the CNF features are not the load-bearing part of the claimed stability. A second check is to replace the CNF densities with ordinary non-contrastive flow densities and see whether the contrast term, rather than density estimation, is what preserves coverage.","supporting_citations":[{"cited_title":"Positive Difference Distribution for Image Outlier Detection using Normalizing Flows and Contrastive Data","cited_arxiv_id":"2208.14024","evidence_quote":"Introduces contrastive normalizing flows and the positive-difference training idea that the paper adapts to signal-fraction estimation."},{"cited_title":"Neyman, Outline of a theory of statistical estimation based on the classical theory of probability, Philosophical Transactions of the Royal Society of London","cited_arxiv_id":null,"evidence_quote":"Supplies the confidence-belt construction used to convert point estimates into coverage-calibrated intervals."}],"review_version":1}