{"id":"14518b8d-d7ea-46f3-ae1e-c64f9d0e9005","arxiv_id":"2506.04491","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GollumFit is a new open-source C++ and Python framework that performs fast binned-likelihood fits over tens of detector and flux nuisance parameters for neutrino telescope data.","lead":"The IceCube Collaboration released GollumFit, an open-source framework that performs binned-likelihood fits on neutrino telescope data using 38 built-in nuisance parameters. It is designed so that neutrino telescope analyses, including future joint searches, can quickly optimize high-dimensional likelihoods over detector and flux uncertainties.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FastMC compression is the load-bearing link: Eq. (6) is asserted, not validated; if meta-event averaging biases reweighted expectations, the claimed accurate high-dimensional fits on compressed sets are unsupported.","rationale":"Reading the paper in good faith: it is a software paper with an open repository, example Monte Carlo, and a believable modular design; the reproducibility bar is met. The central claim, however, includes both 'uniquely incorporates ... parameters' and 'solves ... in a time-efficient manner.' The uniqueness wording is unsupported by comparison with other public frameworks, but that is a marketing overstatement rather than a correctness risk for the software itself. The correctness risk is concentrated in FastMC. The reader's weakest_assumption identifies the same point, and I agree: the manuscript itself flags the k-selection limitation, and the validation is too thin to rule out compression bias. The concern is concrete and testable, and the authors already disclose the limitation, so the appropriate verdict is unchanged (CONDITIONAL): the paper should stand only after Eq. (6) is validated against full-MC fits or the FastMC accuracy caveat is made prominent in the claims. The issue is missing evidence, not suspected misconduct.","tokens_in":16790,"tokens_out":4519,"duration_ms":57379,"concrete_test":"Generate 100 pseudo-experiments: Poisson-fluctuate each bin of the full 13M-event Monte Carlo at nominal and at several perturbed nuisance settings; fit each with the full set (k->0 or k=0.05) and with FastMC at k=0.25, 0.5, 0.75. Compare pull distributions for all 38 parameters and compute the maximum binwise relative violation of Eq. (6) across a grid spanning the prior ranges. A median pull shift above ~0.2 sigma, or Eq. (6) violations above a few percent, would show that compression biases fits and would require a documented k-selection criterion before the accuracy claim can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"GollumFit's time-efficiency claims and the performance benchmarks (Figs. 4-5) all use FastMC, so Eq. (6) in Sec. 3.4 is load-bearing. That equation asserts that the per-bin expectation computed from meta events equals the full-MC expectation for all reasonable parameter values. FastMC constructs meta events by averaging true and reconstructed quantities (Eqs. 4-5) inside true bins of width k*DeltaQ. For the gradient and spline reweighters in Table 1 (domEfficiency, holeiceForward, nuxs, nubarxs, icegrad0-8), the per-event weight is a nonlinear function of these quantities; a sum of nonlinear functions evaluated on individual events is not equal to the count times the function evaluated at the averaged quantities. Eq. (6) can therefore fail even at moderate k. The paper explicitly states that k must be chosen by hand and that a robust determination is future work (Sec. 3.4; Conclusions). The only supporting fit (Fig. 3) is a single self-closure test on FastMC output at k=0.25: the pseudo-data and the fitted model come from the same compressed set, so it cannot detect compression bias, and no full-MC comparison or ensemble is shown. Without such validation, the central claim of accurate, time-efficient recovery of nuisance parameters is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents GollumFit, an open-source C++/Python framework for binned-likelihood analyses in neutrino telescope experiments, together with a set of 38 nuisance parameters spanning atmospheric and astrophysical fluxes, neutrino cross sections, and IceCube-specific detector response. The likelihood is evaluated by reweighting Monte Carlo events, and the FastMC procedure compresses the Monte Carlo set by merging nearby events into meta events. The performance section reports a single closure fit on FastMC-compressed pseudo-data and timing benchmarks on a single machine. The central claim is that GollumFit can accurately and quickly recover input nuisance parameter values in high-dimensional fits.","tokens_in":17040,"tokens_out":7079,"duration_ms":67855,"significance":"If the central claim were fully validated, GollumFit would be a valuable community resource for neutrino telescope analyses: the code is publicly available under LGPL with documentation, a reference Monte Carlo set is included, and the paper is candid about current limitations such as manual tuning of the compression scale and sensitivity to initialization. The likelihood formulation (Eq. 1) and the use of automatic differentiation are standard and sensible. However, the validation is currently too thin to support the claims of accuracy and time-efficiency: it rests on a single self-closure test at one compression level, and the key compression approximation (Eq. 6) is asserted rather than tested. The required additional work is well-scoped and does not require a change in the framework's architecture.","major_comments":[{"comment":"Equation (6) is load-bearing but not validated. The manuscript states that k must be chosen so that the per-bin expectation from meta events equals the full-MC expectation for all reasonable parameter values, but this equality is asserted rather than derived or tested. For the gradient and spline reweighters in Table 1, the per-event weight is a nonlinear function of the true and reconstructed quantities, so the sum of reweighted individual events is not generally equal to the count times the weight evaluated at the weighted-average meta-event quantities; the equality N = sum_e w_e = sum_e-bar w_e-bar holds only at the original weights, not under reweighting. Because Figures 3-5 and the closure test all use FastMC at k=0.25, an unvalidated Eq. (6) directly affects the central accuracy and speed claims. I request a validation study: compare per-bin expectations from FastMC with those from the full 13M-event Monte Carlo set for each parameter varied over, say, +/-2 prior widths, report the maximum and mean per-bin deviations, and show at least one fit with low compression or no compression for comparison.","section":"Section 3.4, Eq. (6)"},{"comment":"The single closure fit shown in Figure 3 cannot detect compression bias because both the pseudo-data and the fitted model are produced from the same FastMC-compressed set at k=0.25. The test verifies internal consistency of the fitting machinery but not the fidelity of Eq. (6). A stronger validation would use an ensemble of pseudo-experiments, report the distribution of best-fit pulls and their coverage, and include at least one fit in which the pseudo-data and/or the model are constructed from the full (uncompressed) Monte Carlo set. Without this, the claim that GollumFit can 'accurately' recover input nuisance parameters is not established.","section":"Section 4, Figure 3"},{"comment":"The timing benchmarks are single runs on a single machine and are not compared with any alternative implementation. Figure 4 shows a linear scaling of per-evaluation time with the number of compressed events, but no comparison with the uncompressed 13M-event set or with an existing binned-likelihood tool, so the claim of being 'time-efficient' is not quantified relative to the available alternatives. Figure 5 reports fit durations between about 1 and 3 hours without run-to-run spread or a stated convergence criterion. I request repeated runs with reported medians and spreads, and, if feasible, a timing comparison with an uncompressed Monte Carlo fit or with a standard reweighting implementation.","section":"Section 4, Figures 4 and 5"}],"minor_comments":[{"comment":"The abstract claims the framework 'uniquely incorporates' the model parameters necessary for neutrino telescopes, but no comparison with existing software is provided; I suggest either softening this wording or supporting it with a short survey of related frameworks.","section":"Abstract and Section 1"},{"comment":"The sentence beginning 'IceCube of nuisance parameters are the efficiency...' appears to be missing words; it should read 'Examples of IceCube nuisance parameters are...'.","section":"Section 2.2"},{"comment":"The phrase 'PhysTools;uses the L-BFGS-B algorithm' contains a stray semicolon; it should read 'PhysTools uses the L-BFGS-B algorithm'.","section":"Section 3.2"},{"comment":"There are several typographical spacing issues, including 'GollumFitis' and 'aDataPaths', and 'respetively' should be 'respectively'.","section":"Section 3.3"},{"comment":"Section 5 contains 'Nontheless' which should be 'Nonetheless', and the Figure 4 caption should say 'as a function of the number of MC events' rather than 'as function'.","section":"Section 5 and Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears intended for a software-oriented journal such as Computer Physics Communications, and the open-source release with documentation is a genuine strength. In my view the central technical concern is the unvalidated FastMC approximation in Eq. (6), which the authors themselves note is selected by hand. If the requested validation against the full Monte Carlo set and the repeated benchmarks are added, I would expect the paper to be suitable for publication; as it stands, the evidence is too limited for the paper's accuracy and time-efficiency claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GollumFit is a genuine open-source artifact: code, docs, example MC set, LGPL. That alone makes it worth peer review. The paper is a clear software description, not a physics claim. What it does well: it bundles the 38 nuisance parameters IceCube analyses actually use, encodes them with gradient/spline/analytical reweighters, and shows a 38-parameter fit converging to input values. The timing benchmarks are honest single-core numbers and the linear scaling on MC size is expected.\n\nThe soft spots are real but not fatal. The 'uniquely' claim in the abstract is unsupported—no search of other frameworks, no comparison to pyhf or other binned-likelihood tools. More importantly, the validation is thin: one closure fit (Fig. 3) on FastMC-compressed pseudo-data at k=0.25. Since both pseudo-data and model come from the same compressed set, it cannot detect compression bias. The stress-test concern about Eq. (6) is correct: for gradient and spline reweighters, the per-event weight is a nonlinear function of true/reco quantities, so averaging those quantities before applying the function is not generally equivalent to averaging the weights. The paper acknowledges k is manual and a robust determination is future work, which is honest, but that means the central performance claim—accurate fits on compressed sets—is not yet established. A full-MC comparison at the same parameters, or an ensemble of fits with different k, would settle it.\n\nI'd send this to peer review. It is a useful, citable software paper for the neutrino telescope community, and the authors have shipped reproducible code and data. The referee should ask for a comparison of FastMC against full MC and a second closure test with k variation, but these are revisions, not grounds for rejection. I would not use the timing numbers as-is for anything quantitative.","headline":"GollumFit is a genuinely useful open-source software artifact whose main physics claim is under-validated by a single self-closure test on compressed MC; worth peer review, with revisions.","tokens_in":17570,"tokens_out":1570,"would_cite":true,"duration_ms":19213,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source fitter tames 38 nuisance parameters for neutrino telescopes","keywords":["binned likelihood","neutrino telescope","IceCube","nuisance parameters","event reweighting","automatic differentiation","FastMC","L-BFGS-B"],"falsifier":"Run the same fit on the same data with two very different FastMC compression scales (e.g., $k=0.1$ and $k=0.5$) and compare the best-fit nuisance parameters: if the per-bin expectations of Eq. (6) are not preserved, the best-fit values will shift by more than the statistical uncertainty.","tokens_in":16571,"feed_emoji":"🔭","tokens_out":3655,"duration_ms":30422,"temperature":0.7,"pith_summary":"This paper presents GollumFit, an open-source framework for binned-likelihood fits to neutrino telescope data. It argues that the framework solves the practical problem of fitting dozens of nuisance parameters simultaneously, and demonstrates that it recovers known input values and runs in minutes to hours on a typical analysis. The key is combining event-by-event reweighting with automatic differentiation and a Monte Carlo compression step called FastMC. If correct, this gives the neutrino community a reusable tool for diffuse-flux and sterile-neutrino analyses that previously required bespoke code.","feed_headline":"Open-source fitter tames 38 nuisance parameters for neutrino telescopes","feed_subtitle":"IceCube's GollumFit recovers true parameter values and cuts fit time with compressed Monte Carlo events.","key_machinery":"The load-bearing object is the per-event weight function $w_e(\\vec{q}_e, \\vec{Q}_e, \\vec{\\theta}, \\vec{\\eta})$ that maps model parameters to the Poisson expectation $\\mu_i = \\sum_{e \\in i} w_e$ in each bin. GollumFit evaluates this weight by three parametrizations (analytical formulas, gradients, or splines), and the likelihood is a modified Poisson (effective) likelihood that accounts for Monte Carlo statistical uncertainty. The optimization uses L-BFGS-B through PhysTools with automatic differentiation for gradients. FastMC then merges events into 'meta events' by binning in true and reconstructed space with a scale factor $k$, preserving the total event number while the number of events shrinks, which speeds up each likelihood evaluation.","core_discovery":"GollumFit's central claim is that a binned-likelihood analysis can incorporate the full set of neutrino-telescope-specific model parameters (38 nuisance parameters covering flux normalization, atmospheric flux shape, cross-section uncertainties, detector efficiency, hole ice, and bulk ice) and still converge to the true parameter values in a feasible time. The paper shows a single fit recovering all nominal nuisance parameters from random initialization, linear scaling of likelihood-evaluation time with Monte Carlo size, and speedups from omitting parameter classes that depend on the weighting method, not simply the number of parameters. The framework bundles nominal IceCube Monte Carlo events as a reference dataset, so analyses can run out of the box.","pith_inferences":["If FastMC compression is validated per analysis, the same reweighting-and-minimization machinery could be applied to other binned-likelihood experiments (e.g., cosmic-ray or gamma-ray observatories) whose systematic uncertainties are encoded as event-weight derivatives.","The modular weighter design suggests that adding new observables (such as azimuth or event morphology) only requires redefining the binning and weighters, so joint analyses across detectors become a matter of sharing splines and priors.","Because the paper only benchmarks parameter recovery on nominal Monte Carlo, a natural stress test is to inject a mismodeled detector response and see whether the 38-parameter fit absorbs it or biases the physics parameters."],"forward_implications":["Fits over tens of nuisance parameters recover nominal values even when initialized randomly away from the truth.","Time per likelihood evaluation scales linearly with the number of Monte Carlo events, and FastMC reduces the event set from about 13 million to about 0.8 million at $k=0.25$, cutting per-evaluation time.","Fit time depends on the weighting method of the parameters, not just their count; omitting six cosmic-ray parameters can be slower than omitting four astrophysical parameters.","The framework is designed to be extendable to other neutrino telescopes and to joint multi-telescope analyses, provided new Monte Carlo and splines are supplied.","FastMC preserves the total expected event number and, when $k$ is chosen correctly, the per-bin expectations for all reasonable model parameters."],"supporting_citations":[{"why":"Supplies the effective likelihood that accounts for Monte Carlo statistical uncertainty in each bin.","marker":"[16]"},{"why":"Provides the LeptonInjector event generator used to produce the bundled reference Monte Carlo events.","marker":"[7]"},{"why":"Provides the analysis context and the subset of Monte Carlo events that GollumFit redistributes as its nominal sample.","marker":"[8]"},{"why":"Defines the DAEMONFLUX calculation that underlies the hadronic-yield and cosmic-ray-spectrum nuisance parameters.","marker":"[14]"},{"why":"Describes the SnowStorm method that motivates the bulk-ice Fourier-mode nuisance parameters.","marker":"[15]"},{"why":"Details the atmospheric density and meson-energy-loss parameters ($\\rho_{\\mathrm{atm}}$, $\\sigma_{\\mathrm{K-Air}}$) and the detector-response parameters DOM efficiency and hole ice.","marker":"[13]"},{"why":"Supplies the L-BFGS-B bound-constrained optimization algorithm used by PhysTools for likelihood maximization.","marker":"[17; 18]"}],"fun_headline_variants":["GollumFit: open-source binned-likelihood for neutrino telescopes","IceCube's GollumFit handles 38 nuisance parameters efficiently","Fast neutrino telescope fits with GollumFit open source","GollumFit: binned-likelihood analysis with many parameters","Neutrino telescope analyses get speed from GollumFit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The compression parameter $k$ must be hand-chosen so that merging Monte Carlo events into meta events preserves each bin's expected count for every plausible value of the model parameters; the paper provides no automatic way to verify this.","fun_headline_variants_meta":{"raw":{"variants":["GollumFit: open-source binned-likelihood for neutrino telescopes","IceCube's GollumFit handles 38 nuisance parameters efficiently","Fast neutrino telescope fits with GollumFit open source","GollumFit: binned-likelihood analysis with many parameters","Neutrino telescope analyses get speed from GollumFit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1587,"prompt_tokens":791,"completion_tokens":796,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":407,"completion_tokens_details":{"reasoning_tokens":705}},"tokens_in":407,"tokens_out":796,"duration_ms":7172,"temperature":1.0,"reasoning_tokens":705,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:40:51.806005+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fit on the same data with two very different FastMC compression scales (e.g., $k=0.1$ and $k=0.5$) and compare the best-fit nuisance parameters: if the per-bin expectations of Eq. (6) are not preserved, the best-fit values will shift by more than the statistical uncertainty.","supporting_citations":[{"cited_title":"Abbasi, et al., LeptonInjector and LeptonWeighter: A neutrino event generator and weighter for neutrino observatories, Comput","cited_arxiv_id":null,"evidence_quote":"Provides the LeptonInjector event generator used to produce the bundled reference Monte Carlo events."},{"cited_title":"Abbasi, et al., Methods and stability tests associated with the sterile neutrino search using improved high-energy𝜈𝜇 event reconstruction in IceCube (5 2024).arXiv:2405.08077","cited_arxiv_id":null,"evidence_quote":"Provides the analysis context and the subset of Monte Carlo events that GollumFit redistributes as its nominal sample."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the SnowStorm method that motivates the bulk-ice Fourier-mode nuisance parameters."}],"review_version":1}