{"id":"45923197-4dca-45ae-afc0-5e985c5682e4","arxiv_id":"2501.05603","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"One trained convolutional network predicts bump significance across mass histograms of different sizes and backgrounds, approaching the accuracy of the ideal likelihood-ratio test.","lead":"BumpNet is a neural network that scans particle-collision mass histograms and flags where a new particle might be hiding, reporting how significant each potential bump is. If it holds up, physicists could automatically screen far more final states than the few percent of LHC searches explored by hand today.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed unbiasedness is conditional on training-distribution coverage; Sec 3.2.2 shows a concrete counterexample where standard and reverse bias transformations cause systematic underestimation of high-mass significance, so the central claim needs a quantitative coverage check or a broader…","rationale":"After reading the paper in good faith, I find the reader's weakest assumption is the correct stress point: the training set defines the method's validity envelope. The paper's own Sec 3.2.2 supplies a counterexample to the 'unbiased' claim, and the failure is not a subtle one: under standard and reverse bias, the network underestimates the true LR significance, concentrated in the high-mass region where real BSM resonances could appear. The proposed test quantifies the boundary of the training envelope in a continuous way rather than relying on the two exaggerated transformations in the paper. I considered other potential concerns. The first-10% exclusion is post-hoc, but the paper applies it consistently thereafter and discloses it. Wider signal widths degrade accuracy, but the network still identifies broader BSM signals in Sec 4.2, and the method's stated purpose is bump identification rather than precise significance. The ATLAS comparisons rely on digitized plots and an approximate resolution convolution, but the agreement is presented at a level consistent with the claimed variance. The GAA false-positive reduction is demonstrated on the same background sample used to develop it, which is a mild overfitting risk, but it is framed as a strategy rather than a calibrated claim. No code is released, which limits reproducibility but does not bear directly on the internal correctness of the argument. None of these displace the training-distribution coverage issue as the load-bearing concern. The paper is a solid proof-of-concept with honest limitations, so the reader's CONDITIONAL verdict stands; the condition should explicitly require either broader training coverage for depleted high-mass tails or a quantitative demonstration that real LHC selections stay within the covered shape manifold.","tokens_in":18080,"tokens_out":4383,"duration_ms":46220,"concrete_test":"Generate a validation set from the DM mother histograms with a continuous family of tail-depletion transforms y'_i = y_i * (1 - lambda * x_i/N_bins) for lambda in {0, 0.25, ..., 1.5}, and inject 1-bin Gaussian signals with Z_LRmax between 2 and 5. For the high-mass half of each histogram, compute the mean DeltaZmax and the fraction of histograms with DeltaZmax < -1, restricting to bins with at least 20 background events to isolate shape bias from Poisson sparsity. If the mean bias exceeds -0.5 sigma for any lambda <= 1.0, the training distribution does not cover a realistic range of tail depletion and the Sec 3.1 unbiasedness claim does not transfer to such LHC-like selections.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a single trained network predicts unbiased bin-by-bin significance across varying histograms, backgrounds, and real data. This holds only insofar as application backgrounds lie in the training manifold. Section 3.2.2 demonstrates a violation: applying the 'standard bias' or 'reverse bias' transformations to DM mother histograms makes BumpNet systematically underestimate the LR significance, with the bias concentrated in the high-mass, low-statistics bins. The paper's own explanation is that training examples predominantly had more background events in the high-mass region, so the network overpredicts the background there. This is exactly the failure mode relevant to real LHC searches: many signal-region definitions (high-HT selections, lepton pT thresholds, invariant-mass tails of highly-boosted objects) produce backgrounds whose high-mass tails are more depleted than the typical DM training histograms. Because the claimed application is scanning O(10^4) histograms, some will fall in this under-covered regime; the network would then under-report the significance of a real high-mass resonance. The unbiasedness shown in Sec 3.1 is therefore a property of the training distribution, not of the method itself. The paper acknowledges the bias and defers a fix to future work, but that is precisely the condition needed for the central claim to transfer to LHC data. This concern is not about disagreement with consensus; it is an internal tension between the broad claim and the paper's own Sec 3.2.2 results.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents BumpNet, a convolutional neural network that maps invariant-mass histograms to per-bin statistical significances for resonant bumps. BumpNet is trained in a supervised manner on a large set of histograms built from eleven analytic background functions and from Dark Machines simulated backgrounds, with Poisson fluctuations and injected Gaussian signals; the targets are likelihood-ratio significances computed from the known underlying signal and background shapes. The network is validated on held-out nominal histograms, on systematically distorted background shapes, on ATLAS H→γγ and di-lepton data, and on Dark Machines BSM signals injected into SM backgrounds. The claimed results are near-zero mean bias (ΔZmax ≈ 0.1σ) with small spread (0.53–0.75σ) on nominal tests, a false-positive rate near 0.1% at 5σ, reproduction of published ATLAS significances, and successful identification of several BSM signals when combined with a Global Analysis Algorithm (GAA) to address the look-elsewhere effect.","tokens_in":18374,"tokens_out":3178,"duration_ms":30636,"significance":"If the central claims hold, BumpNet would be a valuable tool for the data-directed paradigm, enabling rapid scans of the large space of invariant-mass histograms that are not currently examined by dedicated LHC analyses. The paper has real strengths: the training target is an external, well-defined likelihood-ratio significance, so the network is a supervised surrogate rather than a circular fit; the validation includes truly held-out distributions and real ATLAS data; and the application to BSM signals in a realistic simulated sample with O(10^4) histograms is a useful proof of concept. At the same time, the strongest claimed properties — unbiasedness and transfer to arbitrary LHC-like backgrounds — are conditional on training-distribution coverage, and Section 3.2.2 demonstrates a concrete violation of that condition. The paper is honest about this limitation, but the limitation directly affects the paper's central message and needs to be addressed quantitatively before the claims can be taken as established.","major_comments":[{"comment":"The standard and reverse bias transformations produce a systematic negative ΔZmax, concentrated in high-mass, low-statistics bins; the paper attributes this to the training examples predominantly having more background events at high mass and to the impossibility of negative Poisson fluctuations. This is not a peripheral robustness check: it is exactly the regime that can arise in real LHC selections whose cuts deplete the high-mass tail, and the paper's own unbiasedness claim in Sec. 3.1 is stated for the nominal training distribution. The manuscript should either provide a quantitative coverage test (e.g., a measure of how far an application background can deviate from the training manifold before the bias exceeds a stated tolerance) or explicitly restrict the unbiasedness claim to backgrounds that satisfy the training distribution's tail behaviour. A qualitative statement that the bias appears only in sparse bins is insufficient, because the intended application is precisely a scan of many histograms, some of which will fall in this regime.","section":"Sec. 3.2.2, especially Figs. 11 and 12"},{"comment":"The decision to exclude the first 10% of bins from the definition of Zmax is introduced after observing the disagreement in that region and is then applied to all subsequent analysis. Because this exclusion is a post-hoc choice made on the same test set used to report the unbiasedness and false-positive rates, it should be validated as a generalizable procedure rather than a fitted modification. For example, the authors could justify it from the training distribution (e.g., by showing that the first 10% region is underrepresented in training) or demonstrate that the exclusion also reduces bias on an independent test set that was not used to motivate it. As written, the reported false-positive rates and the mean-bias numbers are conditioned on this choice, which weakens the claim that BumpNet is unbiased on untouched histograms.","section":"Sec. 3.1, Fig. 5"},{"comment":"For injected signal widths of 2 and 3 bins, the mean ΔZmax becomes approximately -1.1σ and -1.9σ, respectively, i.e., BumpNet systematically underestimates the likelihood-ratio significance. The paper argues that the primary purpose is identification rather than precise significance, but the network's output is explicitly a significance, and the realistic BSM signals used in Sec. 4.2 are broader than one bin. The manuscript should quantify how this bias propagates into the real-data and BSM-signal results (e.g., whether the 4.5σ Higgs prediction or the >5σ claims in Sec. 4.2 would be reduced if the signal width is not exactly one bin). Without this, the agreement with ATLAS results and the >5σ BSM findings are not fully established for signals that are not perfectly narrow.","section":"Sec. 3.2.3, Fig. 13"},{"comment":"The LEE handling relies on the GAA with several thresholds and tolerances (5σ seed threshold, 5σ family threshold, two-bin mass tolerance, and the requirement of at least two histograms in a family) that are chosen without a systematic optimization or a study of their dependence. The claim that the GAA 'manages' the look-elsewhere effect is based on a single set of choices: it reduces false positives from 25 to 18 in channel 2b and from 28 to 9 in channel 3, but does not eliminate them, and the surviving false positives are disposed of by additional ad hoc observations. Moreover, the W′→qqνν signal is not recovered by the GAA because it appears in only one histogram. The paper should present a more systematic evaluation of the GAA—e.g., false-positive and true-positive rates as functions of its thresholds—or soften the conclusion that the method provides a practical solution to the LEE.","section":"Sec. 4.2, Figs. 19 and 20"}],"minor_comments":[{"comment":"There is a typo: 'As will be shown will be shown' should read 'As will be shown'.","section":"Sec. 3.2.3, last paragraph"},{"comment":"The caption states that the BumpNet prediction is the dashed line and the published significance is the solid line, but the main text describes the opposite assignment. These should be made consistent.","section":"Sec. 4.1, Fig. 16 caption"},{"comment":"The phrase 'aformentioned procedure' contains a typo; it should be 'aforementioned procedure'.","section":"Sec. 2.2.2, 'Smoothing procedure'"},{"comment":"The paper states that 25 and 28 histograms show a significance above 5σ, corresponding to a false-positive rate 'on the order of 0.1%'. For channel 2b, 25 out of 8,104 is 0.31%, which is not 0.1%; please quote the exact rates or state the range.","section":"Sec. 4.2, first paragraph"},{"comment":"The text describes categories with 'exactly two charged leptons' and later subdivides them, but the definition of the parent categories and the role of the Z-candidate veto could be stated more explicitly; a diagram or a precise enumeration of the category tree would aid reproducibility.","section":"Sec. 2.2.2, 'Signal regions'"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution and the honest discussion of limitations is appreciated, but the central unbiasedness claim is materially weaker than the abstract and conclusions suggest because of the Sec. 3.2.2 counterexample and the post-hoc first-10% exclusion. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection. The delayed treatment of the LEE via the GAA also needs more systematic validation before the paper can claim a practical solution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BumpNet is the same group's DDP bump hunter, generalized properly. The new pieces are real: multi-scale convolutional kernels, variable-length histogram handling, dark-Machines simulated backgrounds, a scan of roughly 40,000 histograms with a correlation-based look-elsewhere filter, and validation on the published H→γγ and dilepton significance curves. The training targets are likelihood-ratio significances computed from known injected signals, so the surrogate claim is well defined and not circular. On nominal held-out histograms the bias is near zero and the spread is below 1σ; the agreement with the ATLAS curves is a credible sanity check. The paper is also honest about what it does not cover.\n\nThe main soft spot is exactly what Sec 3.2.2 shows. Standard and reverse bias transformations produce systematic underestimation concentrated in the sparse high-mass tail, and the authors' explanation—the network overpredicts background where training examples had more events—is plausible. That is not a manufactured flaw; it is in the paper, and it matters because real LHC selections often sculpt backgrounds with depleted high-mass tails. The paper acknowledges the bias and defers a fix to future work, but that qualifier sits at the center of the unbiasedness claim. A referee should ask for a quantitative coverage check or a broader training set before the method is used on real data.\n\nTwo smaller issues. The first-10% bin exclusion is introduced after the failure mode was seen, so a prospective rule is needed. And there is no code release, which limits reproducibility for an ML tool. Wide-signal sensitivity degrades as expected; the paper flags that.\n\nOverall, the central idea and validation hold. This is not a finished analysis tool, but it is a solid advance over the earlier proof of concept, and it deserves a serious referee. I'd send it to review and would want the coverage analysis, a prospective exclusion rule, and code and data released in the revision.","headline":"A solid, honest generalization of the DDP bump hunter that deserves a referee, with the caveat that its unbiasedness claim is conditional on training coverage—a point the paper itself makes in Sec 3.2.2.","tokens_in":19004,"tokens_out":3207,"would_cite":true,"duration_ms":30414,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single neural network, BumpNet, turns arbitrary invariant-mass histograms into per-bin resonance-significance maps, matching ideal likelihood-ratio tests within about 0.5–0.8σ and spotting injected new-physics signals across tens of…","keywords":["BumpNet","bump hunting","resonance search","data-directed paradigm","likelihood ratio significance","neural network","invariant-mass distribution","look-elsewhere effect"],"falsifier":"A direct test: take a background-only invariant-mass histogram from a real LHC selection whose kinematic cuts produce a sharply depleted high-mass tail, inject a Gaussian bump of known strength, and compare BumpNet's predicted significance against the exact likelihood-ratio significance. The paper predicts a systematic negative ΔZmax in just those sparse high-mass bins; if the same bias appears on real backgrounds outside the training set, the unbiasedness claim would be falsified in the regime that matters.","tokens_in":17904,"feed_emoji":"📊","tokens_out":5153,"duration_ms":46878,"temperature":0.7,"pith_summary":"BumpNet is a machine-learning replacement for the slow, human-steered part of resonance searches: instead of fitting a predefined functional form to each invariant-mass spectrum, a single trained network turns a smoothly falling histogram into a per-bin estimate of the likelihood-ratio significance of a mass bump. The paper aims to show that this automation works across histograms with different numbers of bins, dynamic ranges, and background shapes—analytic curves, full LHC-like simulations, and real data—with biases close to zero and a spread of roughly 0.5–0.8σ. It also shows that scanning tens of thousands of signal regions at once is feasible, and pairs the network with a correlation-finding algorithm to suppress the look-elsewhere effect. If correct, this extends the data-directed paradigm from a proof of principle into a practical tool that could expand the fraction of LHC final states actually searched for new resonances.","feed_headline":"BumpNet spots mass bumps in 40,000 histograms at once","feed_subtitle":"It predicts per-bin signal significance with roughly 0.5–0.8σ bias and validates on real Higgs and dilepton data.","key_machinery":"The central object is BumpNet, a convolutional neural network with four parallel stacks of kernel sizes 3, 9, 15, and 25, a skip connection that preserves raw bin counts, and a per-bin multilayer perceptron head. It is trained on roughly three million samples built by adding one-bin-wide Gaussian signals to background histograms drawn from eleven analytic functional forms and from smoothed fits to simulated LHC-like events, with the target being the true likelihood-ratio significance computed from the known injected signal and background. The network's job is to compress the background shape and signal location into a per-bin significance value, so that at inference time no fitting or functional-form choice is needed.","core_discovery":"The paper's central claim is that one network, BumpNet, predicts the bin-by-bin statistical significance of a resonant bump in an invariant-mass histogram without knowing the signal or background model, and that these predictions approximate the ideal likelihood-ratio test. On nominal test histograms the difference between the predicted maximum significance and the true one has a central value near zero, with standard deviations of 0.53σ for analytic backgrounds and 0.75σ for simulated LHC-like backgrounds; the false-positive rate for a 5σ threshold is 0.048% and 0.129%, respectively. On real data, BumpNet reproduces the published local significances for the Higgs-to-two-photon distribution, predicting 4.5σ versus a likelihood-ratio significance of 4.2σ, and follows the reported dilepton-resonance significances within its known variance. Finally, when applied to about 40,000 histograms with injected beyond-Standard-Model signals, BumpNet finds each signal at the expected mass and object combination, and the paper's Global Analysis Algorithm removes most of the look-elsewhere false positives.","pith_inferences":["Inference: Because the failure mode identified in the paper is concentrated in sparse high-mass bins under standard and reverse bias distortions, a practical deployment would likely include a per-bin minimum background count cut or a separate network trained on depleted high-mass shapes.","Inference: The same network architecture could be retrained on signal templates other than one-bin Gaussians, such as Breit–Wigner shapes or multi-bin decay chains, extending the approach from bump hunting to general spectral anomaly detection without changing the core likelihood-ratio target.","Inference: The Global Analysis Algorithm's correlation criterion—same mass, same object type, statistically uncorrelated histograms—looks transferable to other broad scans, but its thresholds, such as 5σ seeds and two-bin tolerance, would need to be re-tuned on each dataset before use as a formal discovery procedure."],"forward_implications":["A single trained BumpNet can replace per-analysis background fits for narrow-resonance searches across many final states, since it handles variable bin counts, dynamic ranges down to sparse bins, and background shapes from analytic, simulated, and real sources.","Resonance searches can be carried out on tens of thousands of histograms at once: the paper demonstrates scanning about 40,000 LHC-like histograms and finding injected BSM signals at the expected masses.","The look-elsewhere effect can be tamed without a full trials-factor computation by requiring a family of correlated histograms—same mass, same object combination—to claim an excess, which reduces false-positive histograms from 28 to 9 in the channel-3 background-only scan.","BumpNet's significance predictions are slightly biased for signals wider than one bin, with 2- and 3-bin Gaussian widths showing a growing negative bias, so re-binning histograms to detector resolution is essential for its sensitivity to narrow resonances."],"supporting_citations":[{"why":"The original bump-hunt data-directed paradigm network, whose fixed-bin, narrow-dynamic-range setup BumpNet generalizes and benchmarks against.","marker":"[21]"},{"why":"Provides the likelihood-ratio test statistic used to compute the true per-bin significance targets for training and evaluation.","marker":"[24]"},{"why":"The Dark Machines simulated LHC-like background and signal samples, the source of two-thirds of the training backgrounds and the BSM signal injections.","marker":"[27]"},{"why":"Supplies the published dilepton-resonance background functional form and local significance distributions used to validate BumpNet on real high-mass data.","marker":"[10]"},{"why":"Supplies the H to two photons data points used to show BumpNet reproduces a published discovery-era signal, with a predicted 4.5σ versus a 4.2σ likelihood-ratio significance.","marker":"[33]"}],"fun_headline_variants":["BumpNet AI finds mass bumps in 40k histograms","Neural network predicts signal significance across 40k histograms","BumpNet automates resonance searches, slashing look-elsewhere","AI scans 40k histograms for new particle hints","BumpNet: ML resonance hunter with near-zero bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the background shapes used to train BumpNet, namely eleven analytic curves plus fits to simulated LHC-like events, are representative of the backgrounds real LHC selections will produce; if a real selection depletes the high-mass tail more severely than anything in training, the paper's own systematic-distortion tests show BumpNet will under-predict significance there, and the claimed unbiasedness will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["BumpNet AI finds mass bumps in 40k histograms","Neural network predicts signal significance across 40k histograms","BumpNet automates resonance searches, slashing look-elsewhere","AI scans 40k histograms for new particle hints","BumpNet: ML resonance hunter with near-zero bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000796,"raw_usage":{"total_tokens":3518,"prompt_tokens":977,"completion_tokens":2541,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":2454}},"tokens_in":593,"tokens_out":2541,"duration_ms":15191,"temperature":1.0,"reasoning_tokens":2454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:13:06.947481+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: take a background-only invariant-mass histogram from a real LHC selection whose kinematic cuts produce a sharply depleted high-mass tail, inject a Gaussian bump of known strength, and compare BumpNet's predicted significance against the exact likelihood-ratio significance. The paper predicts a systematic negative ΔZmax in just those sparse high-mass bins; if the same bias appears on real backgrounds outside the training set, the unbiasedness claim would be falsified in the regime that matters.","supporting_citations":[{"cited_title":"A Data-Directed Paradigm for BSM searches: the bump-hunting example","cited_arxiv_id":"2107.11573","evidence_quote":"The original bump-hunt data-directed paradigm network, whose fixed-bin, narrow-dynamic-range setup BumpNet generalizes and benchmarks against."}],"review_version":1}