{"id":"a9330c58-0afc-4098-9907-9d6845a68129","arxiv_id":"2506.11100","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An uncertainty-sampling active learning policy plus a pseudo-streaming workflow trains neutron diffraction structure-finding models with roughly 75% less training data and about 20% shorter training time.","lead":"This paper describes an active learning policy that selects which simulated neutron diffraction patterns to generate next, focusing on areas where the model is most uncertain. It reports that the same structure-finding model can be trained with about 75% less simulated data and about 20% less time on CPU+GPU supercomputers, which reduces a major simulation bottleneck in crystallography.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 75% data-reduction claim lacks a random-sampling control with the same phased, warm-started training protocol, so the improvement may not be due to the uncertainty-based AL policy.","rationale":"The reader correctly flags the unvalidated uncertainty estimates, but the deeper load-bearing issue is that the experimental design never isolates the AL selection rule from the multi-phase, warm-started training protocol. Even a perfectly calibrated uncertainty estimate would not support the central claim unless the AL-selected samples outperform uniformly selected samples under the same training schedule. The paper's baseline is a one-shot bulk uniform sampler with larger dataset sizes, not a same-budget random control with warm-starting and the same per-phase sample allocation. This is a standard and critical control for active learning claims. The concern is addressable: adding a random-sampling phased control and a calibration check would settle whether the 75% data reduction is real and attributable to the proposed policy. The paper has other strengths: six-seed experiments in E1, error bands, a plausible speedup analysis of the streaming workflow, and a public code repository. The missing control does not warrant rejection because the reported accuracy advantage is substantial and the comparison can be repaired. The reader's conditional verdict is therefore appropriate, and no verdict change is needed.","tokens_in":13843,"tokens_out":9584,"duration_ms":103726,"concrete_test":"Run a control workflow identical to the serial AL workflow in Section V-E, except replace the AL sampling distribution p in Eq. (4) with the uniform distribution over the same parameter ranges, using the same phase sizes (13,500 samples per phase), same warm-starting, same epoch counts (400/300/250/200), same seeds, and same validation/test sets. If the random-sampling control's phase-3 classification loss and MSE fall within the AL error bands, the 75% data-reduction claim is not attributable to the AL policy; if the control is significantly worse, the claim is supported. Additionally, to test the uncertainty mechanism directly, compute the rank correlation between σhat^2(S(y);D) and the true squared prediction error on a held-out subset of the study set before each AL phase; a near-zero correlation would indicate that the uncertainty estimates are not driving the selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is the absence of a random-sampling control with the same training protocol. Section V-E compares the AL serial workflow after phase 1 (27k samples) and phase 3 (54k samples) against a baseline that uniformly samples the parameter space and is trained as a one-shot bulk run with larger dataset sizes. The AL model, however, is warm-started from the previous phase and trained with a decreasing epoch schedule (Table II: 400/300/250/200). The reported 4-6x data advantage could therefore be produced by multi-phase warm-starting, by the larger cumulative epoch count, or by the specific validation-based model selection, rather than by the uncertainty-weighted distribution in Eq. (4). Section II asserts that the heteroscedastic uncertainty estimate σhat^2 is strictly positive, bounded, and a faithful proxy for the true error σ^2, but it never validates this assertion. That matters because the entire AL selection mechanism depends on σhat^2 carrying signal. Yet even if the uncertainty estimates are well-calibrated, the current experiments do not isolate the selection policy as the cause of the improvement: a random sampler using the same phased warm-start schedule might achieve the same accuracy with the same data. The streaming results in Section V-F compare serial vs streaming workflows that both use AL, so they cannot validate the AL contribution either.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes a batch-mode active learning (AL) policy based on heteroscedastic uncertainty estimates for training structure-prediction models from simulated neutron diffraction patterns. The policy defines a proposal distribution over lattice parameters proportional to model uncertainty (Eq. 4), and the authors integrate it into a serial phase-based workflow and a pseudo-streaming workflow that overlaps GSAS-II simulation with GPU training. Experiments on three crystallographic symmetry classes compare a four-phase AL serial workflow against a uniform-sampling baseline (Section V-E) and compare serial vs streaming workflows on Polaris and Perlmutter (Section V-F). The central claims are that the AL workflow trains the same model with about 75% less data while improving accuracy, and that the streaming workflow reduces end-to-end training time by about 12-24% with no accuracy loss.","tokens_in":14085,"tokens_out":5582,"duration_ms":54701,"significance":"If the data-reduction claim is valid, the paper is a useful practical contribution: it combines uncertainty-based AL with a streaming HPC workflow for an important scientific inverse problem, and it reports experiments on real GSAS-II simulations with six seeds in E1, open-source code, and two leadership-class platforms. The 4-6x data reduction relative to uniform sampling would be significant for practitioners. However, the current evidence does not isolate the AL selection mechanism from the warm-started, multi-phase training protocol, and the theoretical justification in Section II is only asserted. These issues must be resolved before the central claims can be accepted as stated.","major_comments":[{"comment":"The comparison that supports the 75% data-reduction claim is confounded. The baseline workflow is a one-shot bulk uniform sampler, while the AL serial workflow is trained in four phases with warm-started weights and a decreasing epoch budget (Table II: 400/300/250/200 epochs per phase) and validation-based model selection. The observed advantage of the AL workflow after phase 1 (27k samples) and phase 3 (54k samples) over the baseline could therefore be produced by the phased warm-starting, the larger cumulative number of training epochs, or the model-selection rule, rather than by the uncertainty-proportional distribution in Eq. (4). Please add a control arm that uses the same phase structure, batch sizes, epoch schedule, warm-starting, and validation selection as the AL workflow but draws each phase's samples uniformly (or by random sampling from the same study set). Without such a control, the data-reduction claim is not established.","section":"Section V-E; Section IV-A/IV-B; Table II"},{"comment":"The paper states, 'We then show that a model trained on this new set will have lower total uncertainty,' but no theorem, proof, or citation is provided. This is the core mechanism of the AL policy: sampling from p in Eq. (4) is claimed to reduce sigma^2(D) in Eq. (2). Please provide a precise statement with conditions under which this monotone improvement holds, or reframe Eq. (4) as an unproven heuristic and validate it empirically (e.g., show that selected batches reduce held-out error faster than uniform batches in matched-protocol experiments). The current text overclaims logical support for the policy.","section":"Section II, Eq. (4)"},{"comment":"The heteroscedastic uncertainty estimate sigma-hat^2, which drives Eq. (4), is never calibrated or validated as a proxy for the true prediction error sigma^2 in Eq. (1). The manuscript states a regularity requirement ('strictly positive and bounded') but provides no experiment showing that sigma-hat^2 is informative about actual squared error, particularly in sparse regions of parameter space where AL sampling is most active. The accuracy comparisons in Section V report only classification loss and MSE (Eqs. 6-7), not uncertainty calibration. Please add a calibration analysis - for example, bin held-out predictions by sigma-hat^2 and compare mean sigma-hat^2 with mean squared error, and show that the AL selection actively shifts training mass toward high-error regions.","section":"Section II, III; Eq. (6)"},{"comment":"The E2 scalability results, which are used to support the streaming workflow's 12-24% speedup, are based on a single run per configuration. Without multiple seeds or repeated runs, the reported differences (e.g., 1.24 vs 1.19 speedup across node counts) cannot be distinguished from noise. At minimum, report variance or confidence intervals for E2 or clearly mark these as pilot results. This does not invalidate the E1 finding, but it limits the strength of the generalizability claim in the conclusions.","section":"Section V-F2, Tables IV and V"}],"minor_comments":[{"comment":"Equation (6) has unbalanced parentheses and uses y(x_n) where the model output appears intended as y-hat(x_n); please correct the notation.","section":"Eq. (6)"},{"comment":"The formatting of the E2 parameter ranges is ambiguous: 'a for cubic [3.5, 4.5) [2.5, 5.5)' and 'a and c for trigonal/tetragonal [3.8, 4.2) [3.5, 4.5)' appear as two bracketed intervals with no clear column separation; please restate the ranges in separate rows or columns with explicit parameter names.","section":"Table II"},{"comment":"The conclusion misspells 'diffractrometry' (should be 'diffractometry'), and the abstract contains a subject-verb agreement issue ('The potential ... have been reported'); a light proofread is recommended.","section":"Section VI"},{"comment":"The text says the number of epochs in each phase is approximately inversely proportional to sqrt(N_tot), but Table II lists constant epoch counts (400/300/250/200) rather than values computed from that formula; please clarify how the schedule is derived.","section":"Section V-B; Table II"},{"comment":"The acknowledgments mention an earlier IEEE BigData 2024 paper and a ROSE port; the manuscript should explicitly disclose this relationship and state what new material is presented in this version.","section":"Acknowledgments"}],"recommendation":"major_revision","confidential_remarks":"The core experimental design issue is fixable but central: a matched random-sampling control is needed to support the headline data-reduction claim. The manuscript is also heavily self-referential, with the model M taken from [1] and the uncertainty estimator from [20], both from the same research group; an independent replication or at least systematic comparison against a uniform-sampling phased protocol would substantially increase confidence. The relationship to the IEEE BigData 2024 paper should be disclosed explicitly. Scope-wise, the paper straddles HPC workflows and scientific ML; it may be acceptable for an applied ML audience, but the AL contribution needs sharper comparative evidence before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things up front. The headline claim — training structure-finding models for neutron diffractometry with ~75% less simulated data — is believable but not yet proven to be caused by the active learning policy. The systems work is the strongest part: a carefully measured streaming pipeline with task-level timings on two supercomputers, reported honestly.\n\nWhat is genuinely new: the application, not the machinery. Uncertainty sampling is standard, the heteroscedastic estimator comes from the group's own earlier paper [20], and the Gaussian-mixture interpolation in Eq. (4) is a small trick. Using that combination to steer GSAS-II data generation for structure finding is new, and the evaluation on external-simulator test data is legitimate; the result is not forced by construction. E1 runs six seeds with error bands, which beats the norm in this literature. Code is available on GitHub.\n\nThe soft spots, in proportion. The big one is the missing control: the AL serial workflow is warm-started across phases with a decreasing epoch schedule, while the baseline is a one-shot bulk run with uniform sampling. The 4–6x data advantage could come from the phased training protocol, the larger cumulative epoch count, or the validation-based model selection, rather than from the uncertainty-weighted distribution in Eq. (4). A phased random-sampling control with the same schedule would isolate the AL policy; it should be the condition for acceptance. Second, Section II asserts 'we then show that a model trained on this new set will have lower total uncertainty' — no such proof appears. The paper also never calibrates σ̂² against actual prediction error, and the entire sampling distribution rides on that proxy. Third, E2 is a single run per configuration — the paper acknowledges it — so the scaling numbers are indicative rather than conclusive; that is a minor caveat, as is the scope of three symmetry classes and one material, which the authors also flag.\n\nThe conclusion overreaches when it attributes the savings directly to the AL policy, which is exactly the causal claim the experiments do not isolate. That is a wording fix plus one experiment, not a wall.\n\nThis paper is for people building ML pipelines on simulated diffraction data and for anyone who wants a worked example of AL evaluation pitfalls. It deserves a serious referee: the question matters, the empirical core is cleanly framed, and the gaps are fixable.\n\nRecommendation: engage, send to peer review, and require the phased random-sampling control before acceptance.","headline":"Plausible data-reduction result, but the missing random-sampling phased control means the AL policy is not isolated as the cause; the systems work is the strongest part.","tokens_in":14651,"tokens_out":5810,"would_cite":true,"duration_ms":53687,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An active learning policy trains neutron structure-finding models with about 75% less data.","keywords":["active learning","uncertainty sampling","neutron diffractometry","crystal structure prediction","heteroscedastic uncertainty","streaming training workflow","CPU+GPU pipeline","simulation-based machine learning"],"falsifier":"Run the serial AL workflow and a uniform-random-sampling workflow with the same total number of simulated samples, the same phase schedule, and the same epoch counts; if the random sampler reaches equal accuracy at equal data volume, the AL policy's data-reduction claim is not supported. A second check is to compare predicted $\\hat\\sigma^2(S(y);D)$ against the empirical squared error $\\|y-\\hat y(S(y);D)\\|^2$ on a dense held-out grid: large deviations in sparse regions would indicate the sampling distribution is built on miscalibrated uncertainties.","tokens_in":13599,"feed_emoji":"⚛️","tokens_out":6899,"duration_ms":59591,"temperature":0.7,"pith_summary":"This paper argues that an active learning policy based on uncertainty sampling can train supervised structure-finding models for neutron diffractometry with roughly 75% less simulated training data than a conventional uniform-sampling baseline, while achieving comparable or better prediction accuracy. The motivation is practical: exhaustive grid simulation over unit-cell parameter spaces is the bottleneck, and much of that data is uninformative. The authors further design a pseudo-streaming CPU+GPU workflow that overlaps simulation with training, reporting about 20% shorter total training time without accuracy loss. If these results hold, the combination of fewer simulations and better time-to-solution would make ML-based structure determination much cheaper to deploy for crystallography.","feed_headline":"Active learning cuts neutron-structure training data by 75%","feed_subtitle":"Uncertainty sampling needs a quarter of the simulated data, and streaming trims training time by ~20%.","key_machinery":"The load-bearing object is the sampling distribution over unit-cell parameter space, defined as a prior-weighted Gaussian mixture placed on an interpolated surface of the model's estimated heteroscedastic uncertainty: $p(y) \\propto p_Y(y) \\sum_n \\hat{\\sigma}^2(S(\\bar y_n);D) \\, e^{-(y-\\bar y_n)^2/2\\tau^2}$ over a study set of roughly equally spaced parameters. This distribution redirects simulation effort toward Bragg profiles the current model predicts least well, while the prior $p_Y$ suppresses outliers that are uncertain but unrepresentative. The streaming workflow is a secondary mechanism: it overlaps training tasks on GPUs with simulation tasks on CPUs and uses NUMA-aware CPU/GPU binding to recover otherwise idle resources.","core_discovery":"On the authors' own terms, the central discovery is that querying new simulation points from a distribution proportional to the model's estimated heteroscedastic uncertainty, rather than from a uniform grid, collapses the data requirement for training a multitask classifier-regressor of crystal structure. After four active learning phases, the same model reaches baseline accuracy with about 27,000 to 54,000 training samples where the uniform baseline needs roughly four to six times more, and it does so with lower classification loss and mean squared error. The paper also demonstrates that a streaming workflow, which splits each simulation batch between a serial analysis pipeline and a concurrent simulation task, matches the serial AL workflow's accuracy while cutting wall-clock training time by about 13% to 24% across two platforms and dataset scales.","pith_inferences":["A natural next test, not performed in the paper, is to compare the AL policy against random sampling with identical total data and epoch budgets; such a comparison would isolate how much of the gain comes from query selection rather than from simply training on more data.","The policy's effectiveness presumably depends on the uncertainty estimator being calibrated in sparse regions; a calibration check comparing predicted $\\hat\\sigma^2$ with empirical squared error on held-out parameter regions would tell whether the gains persist outside the tested symmetry classes.","One could extend the study-set interpolation to adaptive spacing, shrinking $\\tau$ or locally refining the study set in high-uncertainty regions, which might further reduce the number of simulated profiles needed.","The streaming design suggests that other three-phase loops of simulate, train, and select could reuse the same overlap pattern whenever simulation and training have different hardware affinities."],"forward_implications":["Structure-finding models for neutron diffraction can be trained with roughly a quarter of the simulated data, shortening the data-generation phase from days to hours for a given accuracy target.","The same model checkpoint reaches lower classification loss and MSE than uniform-grid training at equal or larger dataset sizes, so practitioners can either cut simulation cost or improve model quality at a fixed budget.","A pseudo-streaming workflow that overlaps simulation with training recovers about 13% to 24% of wall-clock time on CPU+GPU systems without changing the final model accuracy.","The authors state that the pipeline carries over directly to X-ray diffractometry and is being applied to other simulation-steered ML workloads such as diffusion-equation surrogates and detector design."],"supporting_citations":[{"why":"Supplies the multitask classifier-regressor network and its performance-tested training scheme that the AL workflow reuses as its model.","marker":"[1]"},{"why":"Provides the terminology and framing for uncertainty-sampling active learning on which the policy is built.","marker":"[19]"},{"why":"Gives the heteroscedastic uncertainty estimate $\\hat\\sigma^2$ used to build the sampling distribution in Eqn. (4).","marker":"[20]"},{"why":"Motivates including the prior $p_Y$ in the sampling distribution to avoid outlier examples with high uncertainty but low representativeness.","marker":"[21]"},{"why":"Supplies the rejection-sampling basis for drawing from the uncertainty-weighted distribution before interpolation is introduced.","marker":"[22]"},{"why":"The GSAS-II simulator that generates every labelled Bragg profile used in training, validation, test, and study sets.","marker":"[23]"}],"fun_headline_variants":["Active learning slashes neutron diffraction training data by 75%","Uncertainty sampling: 75% less data for neutron structure","Active learning needs 75% fewer simulated patterns for diffraction models","Streaming + active learning: 75% less data, 20% faster training","Neutron structure models: active learning cuts data by 75%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole data-reduction result rests on the model's estimated uncertainty $\\hat\\sigma^2$ being a trustworthy proxy for true prediction error in regions the model has seen little data: if the uncertainty estimate is miscalibrated or biased low in sparse regions, the sampler will not consistently choose the most informative next simulations.","fun_headline_variants_meta":{"raw":{"variants":["Active learning slashes neutron diffraction training data by 75%","Uncertainty sampling: 75% less data for neutron structure","Active learning needs 75% fewer simulated patterns for diffraction models","Streaming + active learning: 75% less data, 20% faster training","Neutron structure models: active learning cuts data by 75%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000471,"raw_usage":{"total_tokens":2309,"prompt_tokens":879,"completion_tokens":1430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":1336}},"tokens_in":495,"tokens_out":1430,"duration_ms":9494,"temperature":1.0,"reasoning_tokens":1336,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:58:39.193241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the serial AL workflow and a uniform-random-sampling workflow with the same total number of simulated samples, the same phase schedule, and the same epoch counts; if the random sampler reaches equal accuracy at equal data volume, the AL policy's data-reduction claim is not supported. A second check is to compare predicted $\\hat\\sigma^2(S(y);D)$ against the empirical squared error $\\|y-\\hat y(S(y);D)\\|^2$ on a dense held-out grid: large deviations in sparse regions would indicate the sampling distribution is built on miscalibrated uncertainties.","supporting_citations":[{"cited_title":"Structure Prediction from Neutron Scattering Profiles: A Data Sciences Approach,","cited_arxiv_id":null,"evidence_quote":"Supplies the multitask classifier-regressor network and its performance-tested training scheme that the AL workflow reuses as its model."},{"cited_title":"A Sequential Algorithm for Training Text Classifiers,","cited_arxiv_id":null,"evidence_quote":"Provides the terminology and framing for uncertainty-sampling active learning on which the policy is built."},{"cited_title":"Uncer- 10 tainty Quantification for Deep Learning Regression Models in the Low Data Limit,","cited_arxiv_id":null,"evidence_quote":"Gives the heteroscedastic uncertainty estimate $\\hat\\sigma^2$ used to build the sampling distribution in Eqn. (4)."},{"cited_title":"Active Learning Query Strategies for Classification, Regression, and Clustering: A Survey,","cited_arxiv_id":null,"evidence_quote":"Motivates including the prior $p_Y$ in the sampling distribution to avoid outlier examples with high uncertainty but low representativeness."},{"cited_title":"H ¨ormann, J","cited_arxiv_id":null,"evidence_quote":"Supplies the rejection-sampling basis for drawing from the uncertainty-weighted distribution before interpolation is introduced."},{"cited_title":"GSAS-II: The Genesis of a Modern Open-Source All Purpose Crystallography Software Package,","cited_arxiv_id":null,"evidence_quote":"The GSAS-II simulator that generates every labelled Bragg profile used in training, validation, test, and study sets."}],"review_version":1}