{"id":"92c71025-2f2c-4773-bcce-38a0c7b1ceed","arxiv_id":"2602.10132","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TokaMark defines 14 benchmark tasks on real MAST tokamak data with a hierarchical evaluation protocol and a multi-branch CNN baseline.","lead":"TokaMark is a curated, open benchmark that gives AI models 14 standardized tasks drawn from real MAST tokamak diagnostics, with a baseline model for comparison. It aims to make data-driven fusion plasma models comparable and reproducible across labs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set filtering and variance-normalized NRMSE may make TokaMark scores optimistic: Group 4 tasks can reward predicting the mean, not the event.","rationale":"I agree with the reader's weakest assumption: the evaluation protocol's ability to faithfully measure predictive skill in the target regime is the most load-bearing condition for the central claim. I center on Eq. 2 and the Section 3.2 test filtering because they are internally checkable and directly control the headline numbers. The open-source contradiction is real but contingent—if the repository is released with a commit hash, that objection dissolves. The EFIT-derived target concern for Groups 1-3 is also legitimate, but the paper explicitly labels equilibrium signals as 'derived' (Table 1), and reconstruction benchmarks are legitimate if framed as such; it does not invalidate cross-model comparison. The rare-event NRMSE issue is more fundamental: for the safety-critical Group 4 tasks, the metric can be insensitive to event occurrence, so the published scores may not measure what the task definitions claim. The test-filtering issue compounds this by reporting scores on a clean-window subset. The proposed concrete test—comparing a trivial mean predictor's NRMSE against event-level detection—would settle whether the concern lands. If the mean predictor achieves low NRMSE, the Group 4 results and the 'common measuring stick' claim would need substantial qualification. Since the reader already recommended CONDITIONAL and my concern reinforces it, the verdict should remain UNCHANGED.","tokens_in":17242,"tokens_out":6178,"duration_ms":69072,"concrete_test":"Re-evaluate Group 4 tasks with event-aware metrics. Construct ground-truth event intervals from the test set (e.g., soft X-ray rise, Mirnov burst, Ip quench). Run (a) a trivial predictor that always outputs the per-signal evaluation mean (or the last input sample) and (b) the released baseline. Compute NRMSE per Eq. 2 plus event-level precision/recall/AUC. If the trivial predictor reaches NRMSE < 0.4, or close to baseline, while having zero event-detection skill, then Eq. 2 does not measure the claimed event-prediction capability. Also, run the baseline on an unfiltered test set with missing values at the observed train-set missingness rates; if group NRMSE shifts by >10%, the published clean-window scores are not representative of the incomplete-data regime.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TokaMark's central claim—that its NRMSE scores provide a common measuring stick for plasma models—depends on the evaluation protocol faithfully measuring predictive skill in the sparse, noisy, incomplete-data regime the paper motivates. Two coupled features of the protocol undermine this. First, Section 3.2 filters the test set by requiring full target availability over the prediction horizon and excluding all-NaN input windows, so the held-out evaluation never exercises the missing-data challenge highlighted in Section 2.3. The paper itself says discarding windows with missing components 'wastes valuable examples and can introduce distributional bias,' yet the test protocol does exactly that. Consequently, the benchmark cannot substantiate its 'robustness to incomplete state information' objective, and the reported scores describe a cleaner distribution than the motivating use case. Second, Section 3.3 / Eq. 2 normalizes RMSE by the signal's empirical standard deviation sigma_k computed over evaluation shots. For Group 4 rare-event signals—soft X-ray, Mirnov, plasma current—sigma_k is dominated by quiescent, non-event periods. A model that always emits the mean or a slow baseline can achieve NRMSE well below 1 without ever predicting an event. Thus Task 4-1's NRMSE ≈ 0.344 does not by itself demonstrate early-warning skill, and the group-level comparison is not a valid measure of MHD/detection capability. These are internal inconsistencies in the benchmark's own definitions, not disagreements with external consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TokaMark, a benchmark suite for evaluating AI models on real MAST tokamak data. It defines 14 standardized tasks in 4 groups (equilibrium reconstruction, magnetics dynamics, profile dynamics, and long-horizon forecasting of MHD/rare events), packages 39 diagnostic/actuator signals from 11,573 FAIR-MAST shots, and provides a shot-level split, window-based data loading, a hierarchical evaluation protocol with NRMSE aggregation, and a released multi-branch convolutional encoder–decoder baseline. Baseline results show low NRMSE for Groups 1–2 (0.163 and 0.126) and higher for Groups 3–4 (0.339 and 0.476).","tokens_in":17481,"tokens_out":5598,"duration_ms":56302,"significance":"If the evaluation protocol is valid, TokaMark would be a valuable community resource: it is openly released, includes standardized task definitions, a reproducible baseline, and a clean shot-level split, which are exactly what the fusion-ML community lacks. The shipment of code, data loaders, and baseline model is a concrete strength. However, the benchmark's central claim—that its scores provide a reliable common measuring stick—depends on the evaluation protocol faithfully measuring the capabilities the paper emphasizes: robustness to missing data and rare-event forecasting. The protocol has load-bearing weaknesses in both areas, so the current results must be treated with caution until those are addressed.","major_comments":[{"comment":"The test-set filtering rule in §3.2 excludes windows with all-NaN inputs and requires all output signals to be fully available over the prediction horizon. This directly contradicts §2.3's statement that 'Naively discarding shots or windows with missing components wastes valuable examples and can introduce distributional bias.' The held-out evaluation never exercises the sparse/incomplete-data regime that the paper motivates as a core challenge ('robustness to incomplete state information' in §1.2, Group 3 in §3.1.3). Consequently, the reported NRMSE values describe performance on a cleaned distribution and cannot substantiate the benchmark's robustness objective. Please either include missing-data windows with appropriate masking or partial-output metrics, or explicitly re-scope the robustness claim and report the fraction of excluded windows.","section":"§3.2 (Data Preparation) and §2.3 (Data-driven Challenges)"},{"comment":"The metric NRMSE_k = RMSE_k / σ_k, with σ_k computed over evaluation shots, is not a valid measure of early-warning skill for Group 4 rare-event tasks. The paper states that 'NRMSE_k = 1 corresponds to a model no better than approximating the signal by its mean'—this is only true for a constant prediction. For signals whose variance is dominated by quiescent periods (soft X-ray, Mirnov, plasma current), a model emitting a slowly varying baseline can achieve NRMSE well below 1 without ever predicting the event. Thus Task 4-1's NRMSE ≈ 0.344 does not by itself demonstrate MHD/early-warning capability. Please add event-focused metrics (e.g., AUC, detection latency, false-alarm rate) for Group 4, and present NRMSE only as a secondary signal-fidelity measure.","section":"§3.3, Eq. (2)"},{"comment":"The baseline for non-Markovian tasks truncates input context to 50 ms ('inputs for non-Markovian tasks are truncated to a duration of 50ms'), while Table 2 specifies 100 ms input windows for all Group 4 tasks, and §3.1.5 argues that long context is essential for these tasks. The baseline therefore does not evaluate the defined tasks; its scores are for a truncated variant. This undermines the 'realistic lower bound' claim in §4.3 for Group 4 and prevents fair comparison with future models that use the full specified context. Please either extend the baseline to accept the full specified windows, or explicitly report the truncation as an architectural limitation and re-state the baseline results as corresponding to that truncated version.","section":"§4.2 (Experimental Settings) vs Table 2"}],"minor_comments":[{"comment":"The arXiv abstract states that the dataset and tooling 'are open-sourced' with a GitHub link, while the body abstract says 'The benchmark, documentation, and tooling will be fully open sourced upon acceptance.' Please reconcile these statements and confirm the public availability status.","section":"Abstract"},{"comment":"The table is poorly formatted in the manuscript text: the row for Task 3-3 is garbled (output window is missing), and the column alignment for Group 3 is unclear. Please fix the table so each task has explicit input and output window values.","section":"Table 2"},{"comment":"Equation (2) defines both RMSE_k and NRMSE_k in a single line, which is confusing. Consider splitting into two equations or adding a sentence clarifying the relationship.","section":"§3.3, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"This is a serious benchmark contribution with real potential value, but the evaluation protocol has flaws that affect the interpretation of the headline results. The issues are fixable within the manuscript's scope: adjust the test set to include missing-data cases, add event-based metrics for Group 4, and align the baseline with the task definitions. I recommend major revision rather than reject. The authors should also be asked to verify the 'first large open benchmark' claim against other recent releases, though that is a presentation concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TokaMark is the kind of infrastructure the fusion-ML community actually needs: a curated set of 14 tasks on the open FAIR-MAST data, a shot-level split, a released baseline, and a hierarchical evaluation scheme that makes sense for both single-task and foundation-model comparisons. The task taxonomy is thoughtful, and using real experimental data rather than clean simulations is a definite plus. The baseline numbers, with Group 1-2 NRMSEs around 0.15 and Group 3-4 above 0.3, give a useful first map of task difficulty. Credit where due: this is a concrete, reproducible artifact, not just a proposal.\n\nThe soft spots are real but not fatal. First, the test set filtering in Section 3.2 directly contradicts the paper's own motivation about robustness to missing data. The benchmark says models should handle incomplete signals, then evaluates only on windows where all outputs are fully available. That means the scores describe a cleaner distribution than the motivating use case. This is easy to fix by reporting a second result on an unfiltered test set or at least clearly acknowledging the limitation. Second, the NRMSE normalization in Eq. 2 is problematic for Group 4 rare-event tasks. Since sigma_k is dominated by quiescent periods, a model that just tracks the slow baseline can achieve NRMSE below 1 without ever predicting the event. Task 4-1's 0.344 may indicate real skill, but the metric alone doesn't demonstrate early-warning capability. The paper should add event-focused metrics (recall, time-to-event, or similar) or at least discuss this distortion.\n\nAnother issue worth raising: the open-source status is contradictory between the arXiv abstract (says open-sourced with a URL) and the paper's own abstract (says will be open-sourced upon acceptance), and there is no commit hash. That matters for reproducibility. Also, the reported scores have no error bars, and overlapping windows from the same shot are aggregated as if independent. The shot-level split prevents train/test leakage, but the correlated windows can make the variance estimates optimistic. On the circularity concern about EFIT-derived targets: yes, Groups 1-3 are partly surrogate reconstruction, but the paper is transparent about this, and it is a known property of using derived quantities. I would not treat it as a load-bearing flaw.\n\nOverall: this deserves a serious referee. The authors need to clarify the open-source commitment, justify or change the test filtering, and add a discussion of the rare-event metric. None of these are deep conceptual problems; they are protocol choices that can be addressed in a revision. If the code and data are actually available, this could become a standard benchmark for fusion-ML. I would bring it to a reading group and would cite it in my own work once the artifacts are confirmed public.","headline":"TokaMark is a genuinely useful benchmark artifact that deserves serious refereeing, but the evaluation protocol has fixable soft spots in missing-data handling and rare-event metrics.","tokens_in":18140,"tokens_out":3062,"would_cite":true,"duration_ms":32906,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TokaMark gives AI tokamak models a common benchmark on real data","keywords":["tokamak","plasma modeling","benchmark","MAST","data-driven fusion","NRMSE","multi-modal diagnostics","disruption prediction"],"falsifier":"Run a trivial mean-predictor on the Group 4 targets: if it achieves NRMSE close to or below the reported baseline on any task, the headline numbers for that task do not demonstrate predictive skill. The benchmark could also be validated by recomputing scores with event-based metrics (detection rate vs. false alarms) on the same held-out windows; if rankings change materially, the NRMSE protocol is not capturing what Group 4 claims to measure.","tokens_in":17028,"feed_emoji":"🧲","tokens_out":4845,"duration_ms":52310,"temperature":0.7,"pith_summary":"The paper introduces TokaMark, an open benchmark built from real MAST tokamak diagnostic data, with 14 standardized tasks across four difficulty groups, a hierarchical evaluation protocol, and a released multi-branch convolutional baseline. It aims to fix the fragmentation of fusion datasets by giving fusion and machine-learning researchers a shared measuring stick for comparing AI plasma models. If correct, it means models trained on different tokamaks or by different labs can be scored on the same tasks, and the baseline results show which plasma-prediction problems are currently tractable and which are not. The benchmark is open-sourced along with tooling and baseline training scripts.","feed_headline":"Open benchmark ranks AI plasma models on real tokamak data","feed_subtitle":"14 standardized tasks over 11,573 MAST shots give fusion AI a shared yardstick for fair comparison.","key_machinery":"The benchmark itself is the central object: a signal taxonomy (time series, profiles, videos) with origins (diagnostics, actuators, derived), plus a windowing scheme that defines tasks as reconstruction, autoregressive forecasting, or reconstructive forecasting. The evaluation machinery is hierarchical NRMSE aggregation—samples to windows to signals to tasks to shots to groups—where each signal error is normalized by its empirical standard deviation. The baseline is a multi-branch convolutional encoder–decoder with a shared latent vector, using 1D, 2D, or 3D convolutions matched to input modality. This machinery converts the heterogeneity of multi-rate, incomplete tokamak data into comparabl","core_discovery":"TokaMark is the first large, open benchmark for evaluating AI models on real tokamak diagnostic data. The authors curate 39 signals from 11,573 MAST discharges and define 14 tasks organized into four groups—equilibrium reconstruction, magnetics dynamics, profile evolution, and MHD/disruption forecasting—each with standardized input and output windows. Performance is summarized by a normalized root-mean-square error (NRMSE) computed per signal and aggregated hierarchically to tasks and groups. A generic multi-branch convolutional encoder–decoder baseline, trained separately per task, achieves group NRMSEs of 0.163, 0.126, 0.339, and 0.476, with the hardest task (Task 4-5, locked-mode precurso","pith_inferences":["A natural extension the authors leave implicit is adding event-based metrics—detection rate and false-alarm rate—alongside NRMSE for Group 4, since a model that never predicts a rare event can still score near NRMSE=1.","The selective filtering of test windows with incomplete outputs softens the benchmark's missing-data robustness claim; a stress test that deliberately masks random inputs at evaluation time would directly probe that property.","The baseline's fixed 50 ms input truncation for non-Markovian tasks may understate what longer-context models could achieve, so varying context length is a cheap way to test the benchmark's sensitivity to temporal memory."],"forward_implications":["If adopted, TokaMark gives fusion-AI papers a common yardstick: any model can be scored on the same 14 tasks and compared against published baseline numbers.","The baseline's strong Group 1 and Group 2 scores (NRMSE below 0.17) suggest that equilibrium reconstruction and short-horizon magnetics forecasting are practical targets for real-time neural surrogates.","The weak Group 3 and Group 4 scores, including Task 4-5 above 1.0, mark where generic feed-forward architectures currently fail and where physics-informed or temporally structured models would need to improve.","The hierarchical metric isolates which signals drive a task's difficulty, so researchers can tell whether a model struggles with fast magnetics, slow transport, or rare-event precursors."],"fun_headline_variants":["TokaMark: 14 tasks to benchmark tokamak AI","First open benchmark for tokamak AI: 14 tasks, 11,573 shots","TokaMark: unified benchmark for tokamak plasma AI","14 tasks, 11k shots: TokaMark sets the bar for fusion AI"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the normalized RMSE, computed after excluding test windows whose outputs are not fully available, faithfully captures predictive skill in the sparse, noisy regime; for rare-event tasks, most of a signal's variance comes from non-event periods, so a model that never predicts the event can still score near NRMSE=1.","fun_headline_variants_meta":{"raw":{"variants":["TokaMark: 14 tasks to benchmark tokamak AI","First open benchmark for tokamak AI: 14 tasks, 11,573 shots","TokaMark: unified benchmark for tokamak plasma AI","14 tasks, 11k shots: TokaMark sets the bar for fusion AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000942,"raw_usage":{"total_tokens":3899,"prompt_tokens":815,"completion_tokens":3084,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":3001}},"tokens_in":559,"tokens_out":3084,"duration_ms":21278,"temperature":1.0,"reasoning_tokens":3001,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:04:42.235797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a trivial mean-predictor on the Group 4 targets: if it achieves NRMSE close to or below the reported baseline on any task, the headline numbers for that task do not demonstrate predictive skill. The benchmark could also be validated by recomputing scores with event-based metrics (detection rate vs. false alarms) on the same held-out windows; if rankings change materially, the NRMSE protocol is not capturing what Group 4 claims to measure.","supporting_citations":[],"review_version":1}