{"id":"272acbbf-3aec-4659-8188-fed5c06633a0","arxiv_id":"2509.04360","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MMM overpredicts confinement in most NSTX discharges, agrees better at high beta and broad profiles, and its electron temperature prediction with ion temperature fixed is matched by simply setting Te = Ti.","lead":"This paper tests a fast computer model for plasma turbulence, the Multi-Mode Model, against 37 recorded NSTX tokamak experiments to find out where its temperature predictions can be trusted. The result is a useful map of the model's biases: it usually predicts temperatures that are too steep and hot, and its best electron-temperature result is no better than a trivial rule of thumb.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unquantified experimental profile errors leave the regime-dependence correlations of MMM's Te error (Figs. 12-14, Table I) vulnerable to systematic artifacts.","rationale":"The reader's weakest_assumption—that the experimental profiles are unbiased across parameter space—is indeed the most load-bearing condition for the paper's central claims. The paper's quantitative conclusions are entirely based on comparisons of MMM-predicted profiles to experimental fits (e.g., RMSE values in Table I, correlation coefficients in Figs. 12-13). Without quantifying the uncertainties in those fits, the reported trends could be produced by systematic errors that correlate with the very parameters (beta, tau_E, peaking) the paper claims drive MMM's performance. This is not a manufactured concern: the authors themselves admit the errors are unquantified (Sec. II). The concern affects both the regime-dependence claims (Sec. V) and the Te=Ti baseline comparison (Sec. VI), which is the paper's most provocative result. The reader correctly identified this and assigned a CONDITIONAL verdict; my analysis does not change that verdict, hence UNCHANGED. I considered other potential issues (e.g., the NSTX calibration of the ETG model, the outlier removal for the tau_E correlation) but the measurement-error issue is more fundamental because it undermines the entire empirical basis of the comparisons. The proposed concrete test—Monte Carlo propagation of profile uncertainties—would directly settle whether the correlations and model-baseline differences survive realistic error perturbations. This is a feasible, well-defined check that addresses the weakest link without requiring new experimental data.","tokens_in":34772,"tokens_out":5866,"duration_ms":55602,"concrete_test":"Propagate the experimental profile uncertainties into the analysis: for each discharge, sample Te and Ti profiles from their measurement/fitting covariances (or, if these are unavailable, from a conservatively estimated 10-20% systematic error), recompute the RMSE metrics and the correlation coefficients in Fig. 12 and Table I over many Monte Carlo trials. If the rank correlations with beta/tau_E or the difference between MMM Te-only and Te=Ti baseline errors lose significance (e.g., 95% CI crossing zero or overlapping), the paper's regime-dependence and baseline-comparison claims are artifacts of the unquantified errors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II explicitly states that 'Measurement errors in the raw NSTX data and systematic errors in profile fitting methods increase the uncertainty in these comparisons, though they are not quantified here.' The paper's central quantitative claims—MMM overpredicts confinement (28±13% Te RMSE), and the trends with beta, tau_E, and peaking (Secs. IV-V)—all use these experimental profiles as ground truth without error bars. If fitting errors correlate with beta, peaking, or confinement time (e.g., edge diagnostics are noisier in high-beta discharges, or smoothing flattens profiles differentially), the reported correlations (e.g., r=-0.7 with tau_E after removing two outliers) could reflect the error structure rather than MMM physics. The same issue threatens the Te=Ti baseline comparison in Sec. VI, where the 14±8% vs 11±6% difference could be within experimental uncertainty. Because the paper provides no error estimates on the RMSE values or the correlations, there is no way to assess whether the central sensitivities are statistically robust.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports time-dependent TRANSP simulations in which the reduced transport model MMM predicts Te and Ti profiles for 37 well-analyzed NSTX discharges, plus a second set of 55 discharges used for dedicated beta/nu* scans. It characterizes the MMM transport channels (ETG, Weiland, MTM), finds ion transport mostly neoclassical, demonstrates insensitivity to start time, and identifies correlations between Te prediction error and profile peaking, beta, tau_E, and nu*. It also compares simultaneous Te+Ti prediction with Te-only prediction and with a Te=Ti baseline. The central claims are that MMM generally overpredicts confinement, that agreement improves at higher beta/longer tau_E/broader profiles, and that knowledge of Ti makes MMM's Te prediction comparable to a trivial baseline.","tokens_in":35092,"tokens_out":5484,"duration_ms":55161,"significance":"If the quantitative sensitivities hold, this is a valuable validation and regime-map for MMM, especially for planning NSTX-U predictive simulations. The paper's strengths include a large, well-documented discharge database; fixed TRANSP/MMM settings with no fitting performed in this work; transparent RMSE/median metrics; Spearman checks; an explicit discussion of confounding correlations; and an honest baseline comparison. The full TRANSP run IDs support reproducibility. The main limitations are that the ETG component was calibrated on NSTX data (so part of the agreement is in-sample in the device dimension), the experimental profile errors are explicitly not quantified, and several key subgroup thresholds and outlier removals are post hoc. The paper is therefore a useful sensitivity study, but the quantitative regime-dependence claims need robustness analysis before they can be taken as definitive.","major_comments":[{"comment":"The paper states that measurement errors and systematic profile-fitting errors are \"not quantified here.\" All central accuracy claims—median Te RMSE 28±13%, the r=-0.53/-0.7 correlations with beta_e and tau_E, and the 14±8% vs 11±6% baseline comparison—are differences between MMM and these same experimental profiles. If fitting errors correlate with beta, peaking, or confinement time, the reported regime dependences could be artifacts. Please supply at least a sensitivity analysis with plausible profile-error envelopes and confidence intervals on the RMSE values and correlation coefficients, or explicitly relabel the quantitative results as relative comparisons subject to unknown systematic error.","section":"Sec. II, Figs. 12–14, Table I"},{"comment":"The strongest correlation (r=-0.70, rank -0.75) is obtained after \"removing two outlier points which TRANSP calculated unphysically large tau_E for even in the fully interpretive runs.\" No reproducibility criterion is given for this removal, and tau_E is itself correlated with beta_e and profile peaking, as the paper notes. Please report all 37 points, show the effect of the removal on r, define \"unphysical,\" and provide uncertainties (e.g., bootstrap) for the correlations. A partial-correlation or multi-variable analysis is needed to separate tau_E from beta/peaking; without it, the \"most robust correlation\" claim is not fully supported.","section":"Sec. V A, Fig. 12(b)"},{"comment":"The beta_e=11% and nu*=0.23 thresholds are chosen post hoc because they divide the discharge set into roughly equal groups, and the text states they are \"not physically significant.\" The claim that MMM's electron transport transitions from ETG-dominated to mixed/Weiland-dominated regimes is based on these arbitrary splits. Please show the continuous dependence of transport fractions on beta_e and nu*, or test sensitivity to threshold placement, and quantify the spread within each group. The same issue applies to the beta_p binning in Fig. 3.","section":"Sec. II A, Fig. 2"},{"comment":"The conclusion that Te-only MMM is not better than setting Te=Ti rests on 14±8% vs 11±6% in the text (11±5% in Table I). Without uncertainty estimates or a paired statistical test, these two values may be statistically indistinguishable—which would support the paper's point, but in the opposite direction it also means MMM's improvement over simultaneous prediction is not demonstrated. Please provide per-discharge paired comparisons, confidence intervals on the medians, and a test of whether the Te-only distribution differs from the Te=Ti baseline. This is load-bearing because it tempers the practical value of the Te-only simulations.","section":"Sec. VI, Table I"},{"comment":"The dedicated scans yield r=-0.2 to -0.3 for beta and r=-0.22 for nu*, trends described as weak. The claim that MMM is \"more sensitive to beta than nu*\" is based on differences between correlations whose uncertainties are not reported; with N~55 these differences may not be statistically significant. Please provide confidence intervals and/or a combined regression with beta and nu*, and state explicitly whether the difference in correlation strengths is significant. Otherwise the conclusion exceeds the evidence.","section":"Sec. V B, Fig. 13"}],"minor_comments":[{"comment":"The Te=Ti baseline error is 11±5% in Table I but 11±6% in the main text. There is also a typo: \"the choice of using Te = Te as a baseline comparison\" should be Te = Ti.","section":"Table I / Sec. VI"},{"comment":"\"Inverse aspect ratio of around 1.4\" should likely be \"aspect ratio of around 1.4\" (or inverse aspect ratio ~0.7), given NSTX parameters.","section":"Sec. I"},{"comment":"The data repository placeholder \"(placeholder)\" should be replaced with the actual ARK before publication.","section":"Acknowledgments"},{"comment":"The claimed weak correlation between stored-energy coefficient of variation and Te RMSE is not quantified; please add the correlation coefficient and sample size.","section":"Fig. 7"},{"comment":"The parenthetical \"duration ∼200 ms versus 20 ms\" is confusing; the text elsewhere compares 20 ms time slices against the ~200 ms windows. Please rephrase.","section":"Sec. V B"},{"comment":"Minor typographical issues: \"veraged\" in the Fig. 15 caption, inconsistent formatting of \"PT SOLVER,\" and a missing space in \"PT SOL VER.\"","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"This is a solid validation/sensitivity study appropriate for Physics of Plasmas if the uncertainty and robustness issues are addressed. The ETG calibration is fixed and no fitting is done here, so the circularity concern is modest, but the device-level in-sample nature should be acknowledged more explicitly when claiming NSTX-U predictive value. The strongest correlation is vulnerable to post hoc outlier removal, and the experimental-error caveat is acknowledged but not operationalized. I would not require new experiments, but I would require a sensitivity analysis with error envelopes, bootstrap confidence intervals, and clearer outlier/threshold criteria. The companion-paper cross-reference is appropriate; the data placeholder must be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before the companion paper lands: it's the detailed sensitivity follow-up to the MMM-vs-TGLF comparison, and it does what a good validation study should. It runs time-dependent predictive TRANSP with MMM on 37 well-characterized NSTX discharges, shows the model robustly overpredicts Te (28±13% RMSE when predicting Te and Ti together), and maps where the model does better: higher beta, longer tau_E, broader profiles. The new stuff is real: a dedicated set of discharges chosen to decouple beta and nu*, a start-time robustness demonstration, a Te=Ti baseline comparison, and a computational cost breakdown. I especially respect the baseline. When Ti is fixed, MMM's Te-only RMSE (14±8%) is not much better than just setting Te=Ti (11±6%) at zero compute cost. That's a genuinely useful result for anyone planning scenario optimization, and they don't hide it.\n\nThe paper is also careful about its limitations. It explicitly says measurement errors are not quantified (Sec. II), acknowledged in the text. It checks parametric correlations among beta, tau_E, and peaking, and presents rank correlations alongside linear ones. The time-dependent tracking and the insensitivity to start time are convincing.\n\nThe soft spots are real but proportionate. The unquantified experimental profile errors are the big one. The headline correlations (e.g., tau_E, r=-0.7) are computed against experimental profiles that carry no error bars; if fitting noise correlates with beta or peaking, some of the regime-dependence could be artifact. The beta_e=11% and nu*=0.23 thresholds are post hoc, and the strongest tau_E correlation appears only after dropping two discharges. They're transparent about all of this, but it means the quantitative trends should be treated as provisional. The ETG component was calibrated on NSTX data, so part of the agreement is in-sample; they note it but don't dwell on it. Also, the data repository is a placeholder, which is a minor procedural blemish. The Te=Ti result is the sharpest finding, and it's a double-edged sword: it undermines the practical value of Te-only MMM predictions when Ti is known.\n\nWho should read this? Anyone using reduced models for spherical tokamak scenario development, especially NSTX-U and SMART, and anyone doing model validation as a methodological example. It deserves peer review; the study is reproducible in principle, the analysis is honest, and the conclusions, while modest, are useful. The referee should push for error bars on the experimental profiles and a sensitivity check on the subgroup thresholds, but the work is not fatally flawed. I'd bring it to a reading group and would cite it in my own validation work. Serious thinker: yes.","headline":"A solid, unusually honest validation study of MMM on NSTX discharges that maps regime-dependent biases, though unquantified experimental errors keep the main correlations from being fully secure.","tokens_in":811,"tokens_out":878,"would_cite":true,"duration_ms":29212,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":["52.25.Fi","52.55.Fa"],"model":"deepseek-v4-flash","headline":"The Multi-Mode Model, run time-dependently on 37 NSTX discharges, generally overpredicts electron and ion temperatures, giving overly steep profiles; agreement improves at high beta, long confinement time, and broad profiles, and with Ti kn","keywords":["Multi-Mode Model","NSTX","TRANSP","turbulent transport validation","temperature profile prediction","spherical tokamak","confinement overprediction","beta scaling"],"falsifier":"A direct test: take a new set of NSTX or NSTX-U discharges with independently fitted profiles and quantified error bars, run the same MMM/TRANSP workflow, and check whether (i) the Te error still correlates with beta, tau_E, and peaking, and (ii) Te-only MMM still fails to beat the Te=Ti baseline. If the correlations disappear or MMM clearly beats the baseline once Ti is known, the paper's central claims are falsified.","tokens_in":34741,"feed_emoji":"⚛️","tokens_out":7319,"duration_ms":66243,"temperature":0.7,"pith_summary":"The paper sets out to map where the Multi-Mode Model (MMM), a fast reduced model of turbulent plasma transport, can be trusted to predict temperature profiles in spherical tokamaks. Using time-dependent TRANSP simulations of 37 well-analyzed NSTX discharges, it establishes a consistent tendency: MMM produces too little transport, so it overpredicts electron and ion temperatures and makes the profiles too steep. The error is not uniform—agreement improves for plasmas with higher beta, longer energy confinement times, and spatially broader electron temperature profiles, and it is worse for peaked, low-beta, low-q cases. The most pointed finding is about added value: if the ion temperature is already known experimentally, MMM's electron temperature error (14%) is barely better than simply setting Te = Ti (11%), a computation-free guess. The authors frame the work as a validation map for NSTX-U scenario planning.","feed_headline":"MMM transport model overpredicts NSTX temperatures","feed_subtitle":"Across 37 discharges it predicts too-stiff profiles; with Ti fixed, its Te accuracy matches the free Te=Ti guess.","key_machinery":"The carrying object is MMM, a reduced multi-mode turbulent transport model whose total diffusivity is the sum of four submodel predictions: an electromagnetic electron-temperature-gradient (ETG) model, a microtearing-mode (MTM) model, the Weiland model (ITG/TEM/KBM), and a drift-resistive-inertial ballooning model that is disabled for NSTX. Within TRANSP's PT-SOLVER the summed diffusivities evolve Te and Ti; the collisional ion-electron exchange term Q_ie couples the two channels, so predicting both profiles makes the result sensitive to the composition of transport. The paper's diagnostics are the relative profile offset/RMSE figures of merit and correlations of those errors with beta, coll","core_discovery":"On the paper's own terms: predictive TRANSP runs with MMM on a large, deliberate NSTX database show that the model systematically overpredicts confinement, reproducing temperature levels only when the plasma already has good confinement, broad profiles, and high beta. Electron transport in MMM shifts from ETG-dominated at low beta and high collisionality to a mix with Weiland-mode (ITG/TEM/KBM) transport at high beta and low collisionality, while microtearing modes matter mainly near the edge; the ion channel is almost purely neoclassical. Because collisional energy exchange couples Te and Ti, fixing Ti to experiment changes not only Ti but reduces Te error by half, to 14%, which is statisti","pith_inferences":["The strong correlation between Te error and experimental tau_E suggests MMM's bias may be a fixed under-prediction of transport that only shows up where tau_E is short; testing MMM on a database spanning conventional tokamaks with the same baseline comparison would show whether this is generic or NSTX-specific.","The Te=Ti baseline working so well is a consequence of balanced beam heating in NSTX; in RF-heated or electron-heated regimes the trivial baseline would differ, so MMM's marginal value should be re-measured there rather than assumed.","The worst outliers at low beta and low q hint that the missing transport may be from low-n MHD or fast-ion-driven instabilities, both absent from MMM; the paper's proposed ad hoc edge diffusivity enhancement for flat-Te discharges is a directly testable extension.","The finite-memory result (predictions forget initial conditions after about an energy confinement time) implies that time-dependent validation windows shorter than tau_E cannot distinguish model memory from input driving."],"forward_implications":["MMM scenario scans for NSTX-U can be run with confidence in their time dynamics—predictions converge to the same final profiles regardless of start time—but absolute temperatures should be read with an overprediction bias.","High-beta, long-tau_E, broad-profile NSTX-U scenarios are the ones MMM is most likely to predict reliably; low-beta, low-q, peaked profiles are the danger zone.","For Te-focused transport studies, pinning Ti to measured values and predicting only Te is the cheaper and more accurate workflow—roughly an order of magnitude faster with half the Te error.","Validation claims for MMM that predict Te with Ti known should be compared against trivial baselines; a Te=Ti heuristic already achieves comparable agreement in these beam-heated NSTX discharges.","Since Te prediction error is uncorrelated with which submodel dominates, no single mode in MMM can be blamed for the bias."],"supporting_citations":[{"why":"Supplies the same 37-discharge NSTX set and the earlier MMM-vs-TGLF comparison that this study extends.","marker":"[1]"},{"why":"Defines the physics basis of MMM as a summed multi-mode transport module.","marker":"[39]"},{"why":"Documents the improved MMM implementation used for the predictive runs.","marker":"[40]"},{"why":"Provides the ETG transport model with NSTX calibration, the dominant core electron transport channel at low beta.","marker":"[43]"},{"why":"Provides the microtearing mode model responsible for edge electron transport.","marker":"[44]"},{"why":"Provides the Weiland model (ITG/TEM/KBM) that becomes significant at high beta and low collisionality.","marker":"[45]"},{"why":"Previous MMM modeling of NSTX that informs the flow-shear and DRIBM-exclusion choices.","marker":"[47]"},{"why":"Describes TRANSP/PT-SOLVER, the time-dependent predictive transport solver carrying the simulations.","marker":"[55]"},{"why":"NCLASS neoclassical transport solver that determines the dominant ion transport channel.","marker":"[66]"},{"why":"Supplies the baseline-comparison methodology used to evaluate MMM's Te prediction against the Te=Ti heuristic.","marker":"[79]"}],"fun_headline_variants":["MMM overestimates NSTX temperature profiles","NSTX test: MMM runs hot on temperature","MMM overpredicts NSTX temperature profiles","Model runs hot: MMM overshoots NSTX Te","NSTX predictive test: MMM overconfines"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The experimental temperature profiles used as the standard of comparison are assumed to be unbiased, but the paper does not quantify measurement or profile-fitting errors; if those errors correlate with beta, profile peaking, or confinement time, the reported regime trends could be artifacts rather than properties of MMM.","fun_headline_variants_meta":{"raw":{"variants":["MMM overestimates NSTX temperature profiles","NSTX test: MMM runs hot on temperature","MMM overpredicts NSTX temperature profiles","Model runs hot: MMM overshoots NSTX Te","NSTX predictive test: MMM overconfines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000528,"raw_usage":{"total_tokens":2449,"prompt_tokens":877,"completion_tokens":1572,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1491}},"tokens_in":621,"tokens_out":1572,"duration_ms":12218,"temperature":1.0,"reasoning_tokens":1491,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:13:03.161402+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: take a new set of NSTX or NSTX-U discharges with independently fitted profiles and quantified error bars, run the same MMM/TRANSP workflow, and check whether (i) the Te error still correlates with beta, tau_E, and peaking, and (ii) Te-only MMM still fails to beat the Te=Ti baseline. If the correlations disappear or MMM clearly beats the baseline once Ti is known, the paper's central claims are falsified.","supporting_citations":[{"cited_title":"Rafiq, A","cited_arxiv_id":null,"evidence_quote":"Defines the physics basis of MMM as a summed multi-mode transport module."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the improved MMM implementation used for the predictive runs."},{"cited_title":"Rafiq, C","cited_arxiv_id":null,"evidence_quote":"Provides the ETG transport model with NSTX calibration, the dominant core electron transport channel at low beta."},{"cited_title":"Rafiq, J","cited_arxiv_id":null,"evidence_quote":"Provides the microtearing mode model responsible for edge electron transport."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Weiland model (ITG/TEM/KBM) that becomes significant at high beta and low collisionality."},{"cited_title":"Rafiq, C","cited_arxiv_id":null,"evidence_quote":"Previous MMM modeling of NSTX that informs the flow-shear and DRIBM-exclusion choices."},{"cited_title":"Pankin, J","cited_arxiv_id":null,"evidence_quote":"Describes TRANSP/PT-SOLVER, the time-dependent predictive transport solver carrying the simulations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NCLASS neoclassical transport solver that determines the dominant ion transport channel."},{"cited_title":"Abbate, E","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline-comparison methodology used to evaluate MMM's Te prediction against the Te=Ti heuristic."}],"review_version":1}