{"id":"fad67954-4177-420b-8da6-2a7e87cc0896","arxiv_id":"2602.15084","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multi-modal transformer pretrained on MAST tokamak data beats the TokaMark CNN baseline on 13 of 14 tasks, and warm-start fine-tuning beats from-scratch training on the hardest forecasting and equilibrium tasks.","lead":"TokaMind is a new open-source AI model that learns plasma behavior from many different tokamak sensors at once, then adapts quickly to reconstruction and forecasting tasks. It beats a standard CNN baseline on 13 of 14 benchmark tasks, and warm-start fine-tuning beats training from scratch on the hardest forecasting and equilibrium tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pretraining split unspecified: if pretraining includes TokaMark test shots, warm-start gains could be leakage, not transfer; this is unaddressed and checkable.","rationale":"The reader's verdict is CONDITIONAL, and my analysis does not change that label, but it identifies a different root concern. The reader focused on the CNN baseline being underpowered and the protocol potentially shaped by author overlap. My concern is sharper: the paper never states whether pretraining uses only the TokaMark training split. If it includes test shots, the entire evaluation is compromised, including the warm-start-vs-scratch comparison, which is the most direct evidence for transferable representations. This is not a matter of baseline strength; it is a potential data-leakage flaw. Because the released code could settle it, the condition should be revised to require explicit disclosure and verification of the pretraining split. The reader's call for seed variance and independent baselines remains valid but is secondary. Thus I keep the CONDITIONAL verdict, but the decisive condition is now pretraining/test separation.","tokens_in":17451,"tokens_out":6521,"duration_ms":69832,"concrete_test":"Inspect the released pretraining data loader / configuration (github.com/UKAEA-IBM-STFC-Fusion-FMs/tokamind) to determine whether any shots or windows from the TokaMark test splits are used for pretraining. If they are, re-run fine-tuning with a pretraining corpus strictly limited to TokaMark training shots and verify whether FT-Base still beats Scratch and CNN on the test split. If the code is unavailable, request an explicit statement of the pretraining split and require a repeat of the core experiments with a held-out test set never seen during pretraining.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that fine-tuned TokaMind outperforms the CNN baseline and that warm-start beats scratch—depends on a clean separation between pretraining data and test data. §6.1 describes pretraining on a 'broad signal set' with all benchmark inputs, actuators, and outputs, but never states whether this set is restricted to the TokaMark training split. §6.2 says evaluation follows TokaMark's 'data splits, and evaluation protocol,' but this governs downstream tasks, not pretraining. If pretraining consumes all MAST shots (or any windows from the test split), the fine-tuned model has already seen test-domain signals during the reconstruction objective. The from-scratch and CNN baselines have not, so the reported gains could reflect memorized representations of the test distribution rather than transferable plasma dynamics. This is the single most load-bearing gap because it directly threatens the causal interpretation of the headline results. It is also easily checkable: the released training code or configuration should reveal the exact shot/window selection for pretraining. The paper's silence on this point is a concrete omission, not a matter of 'strong vs. weak baseline.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TokaMind, a multi-modal transformer foundation model for tokamak plasma data, pretrained on the public MAST dataset with a reconstruction objective over DCT3D-compressed chunks. The model is evaluated on the 14 tasks of the TokaMark benchmark, comparing two fine-tuned variants (FT-Base, FT-Tiny) and a from-scratch model against the benchmark's CNN baseline. The main claims are that fine-tuned TokaMind outperforms the CNN baseline on all but one task and across all four benchmark groups, and that warm-start fine-tuning is most beneficial on demanding long-horizon/high-frequency tasks, supporting the value of multi-modal pretraining.","tokens_in":17746,"tokens_out":4831,"duration_ms":51324,"significance":"If the claims hold, this is a valuable open contribution: a lightweight (<10M parameter) schema-flexible transformer, a training-free DCT3D codec with an energy-preservation guarantee, and a reproducible evaluation with released code and weights. The comparison against a matched from-scratch architecture under the same downstream epoch budget is a reasonable way to isolate the effect of pretraining. However, the paper's central empirical claims currently rest on an unspecified pretraining/test split and single-seed point estimates, so the significance is conditional on these being resolved.","major_comments":[{"comment":"The pretraining configuration is not tied to the TokaMark training split. Section 6.1 says pretraining uses 'all benchmark input diagnostics' and 'all benchmark outputs' over 50 ms windows, but it never states whether these windows are drawn exclusively from the TokaMark training shots. Section 6.2 only specifies that evaluation follows TokaMark's splits. If pretraining consumes windows from the test split—or from full-shot windows that overlap test windows—the fine-tuned model has already seen test-domain signals through the reconstruction objective, while the CNN and from-scratch baselines have not. This would make the reported gains reflect test-distribution exposure rather than transferable plasma dynamics. This is the most load-bearing gap because it directly threatens the causal interpretation of the headline result. Please state the exact shot/window selection for pretraining, or","section":"§6.1–6.2 (pretraining split)"},{"comment":"All reported NRMSE values are point estimates from a single run, with no standard errors, confidence intervals, or repeated seeds. Several comparisons that support the warm-start claim are numerically small (e.g., Task 2-2: FT-Base 0.0457 vs Scratch 0.0450; Task 3-2: FT-Base 0.2802 vs Scratch 0.2852; Group 3 average: FT-Base 0.2703 vs Scratch 0.2705). Without seed variance, 'consistent gains' and 'warm-start is generally stronger' cannot be distinguished from optimization noise. Please provide results over at least three seeds with mean ± std (or equivalent), and indicate whether the reported numbers are the best or the mean.","section":"Table 2 and §6.3"},{"comment":"The CNN baseline and the TokaMark benchmark itself are introduced in reference [18], authored by a group that overlaps heavily with the authors of this paper. This is not by itself an error, but it places a burden on the paper to show that the comparison is neutral and that the CNN baseline is strong rather than underpowered. The manuscript does not report the baseline's capacity, training budget, hyperparameter tuning, or whether the same preprocessing and augmentation protocol was used for the baseline. Please add this information, or otherwise provide evidence that the baseline is a competitive reference point, so that 'outperforms the benchmark baseline' is not an artifact of protocol overlap.","section":"§2, §6.1 (baseline independence)"}],"minor_comments":[{"comment":"The conclusion states that TokaMind 'consistently improves over the CNN baseline across all tasks and groups,' but the results in Table 2 show Task 4-5 is not improved. This contradicts §6.3's 'all but one task.' Please rephrase to 'all but one task' consistently.","section":"§7 Conclusions"},{"comment":"The arXiv abstract says training code and model weights 'are publicly available,' while the full text's abstract says they 'will be made publicly available.' Please align these statements and include exact repository/dataset versions.","section":"Title/Abstract vs Full Text"},{"comment":"The statement 'we verified the model can fit a small subset of shots' is presented without evidence. If this is meant to support the claim that residual error is driven by rare regimes, please show the training-loss curve or a small quantitative table.","section":"§6.3, Task 4-5"},{"comment":"The caption uses Δ as 'NRMSE difference to the corresponding DCT3D result,' but the table lists positive Δ values for VAE, which are worse. Consider labeling as 'Δ (VAE − DCT3D)' to avoid ambiguity.","section":"Table 3 caption"},{"comment":"The per-output weights λ_o are introduced but their default values are never specified. Please state whether they are all set to 1 in all experiments, or list the values used.","section":"Eq. (1)"},{"comment":"The y-axis label 'NRMSE vs CNN (%)' is informal. Clarify that positive values mean percentage improvement (lower NRMSE) over the CNN baseline; the current caption could be misread.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The pretraining-split question is easily checkable from the released code, and I recommend making that a condition of acceptance. The paper is otherwise a reasonable empirical contribution, but the overlap between the TokaMind and TokaMark author teams means the benchmark comparison is not fully independent; an external baseline or a clear statement of the baseline's tuning effort would substantially strengthen the paper. Single-seed results are a recurring weakness in this type of study and should be treated as a required fix rather than a preference."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This is a genuinely useful and clearly written empirical paper — to my knowledge the first open-source multi-modal transformer foundation model for tokamak diagnostics, with a parameter-free DCT3D tokenizer that is reproducible and a matched-budget comparison of fine-tuning vs training from scratch. And the central \"pretraining transfers\" claim has a gap the authors can likely patch in one sentence: §6.1 says the pretrained model is trained on all benchmark inputs, all actuators, and all 34 targets, but never states whether those windows are restricted to the TokaMark training split. If pretraining consumed test shots, the warm-start gains could be leakage, not transfer. The \"we follow TokaMark's data splits\" sentence in §6.2 governs downstream evaluation, not pretraining. The stress-test note is right: this is the one detail that makes the headline result interpretable, and it is directly checkable from the released config.\n\nWhat the paper does well: DCT3D is training-free and gives a clean uniform token interface; the schema-flexible chunking and selective freezing are practical; the FT-vs-scratch comparison is the right ablation; and the authors report their own weak spots — Task 4-5 is explicitly flagged, and the x-point_z signal where CNN wins is shown in Table A.1 rather than hidden. The VAE ablation is a nice control showing the cheap default is competitive.\n\nSoft spots, in proportion. Single-seed runs with no error bars is the second real issue: in Table 2, scratch beats FT-Base on several tasks (1-2, 2-2, 3-1), so the \"warm-start helps most where it matters\" claim leans on group averages. The CNN baseline comes from TokaMark, whose author list overlaps heavily with this one; that does not invalidate the comparison, but it tempers \"outperforms the strongest baseline.\" The conclusion overstates by saying \"across all tasks and groups\" when Task 4-5 is worse — the abstract's \"all but one\" is accurate. The availability statements are inconsistent (header abstract: code and weights public with URLs; body abstract: \"will be made publicly available\"). Also note DCT3D's explained variance on the high-frequency coils is 0.63–0.72: Task 4-5 is partly bounded by the codec before the model sees it.\n\nWho it is for: fusion-ML researchers and anyone building domain foundation models over heterogeneous sensor streams. It deserves a serious referee. Send it to review with a short referee list: state the pretraining split, add at least a second seed, fix the overstatement, and reconcile the availability text.","headline":"A solid, clearly written foundation-model paper for tokamak data whose central transfer claim hinges on a pretraining-split detail the paper never states; single-seed and same-team baseline are secondary but real concerns.","tokens_in":18290,"tokens_out":5058,"would_cite":true,"duration_ms":48401,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-modal transformer pretrained on tokamak diagnostics improves reconstruction and forecasting on nearly all benchmark tasks, with warm-start fine-tuning giving the largest gains on the hardest long-horizon, high-frequency problems.","keywords":["Multi-modal Transformer","Foundation Models","Tokamak","Fusion Energy","Plasma Dynamics","DCT3D","Transfer learning"],"falsifier":"Train a CNN baseline with the same parameter count, same total training epochs, and the same input augmentation and dropout used for TokaMind, then re-run all 14 benchmark tasks. If the fine-tuned transformer no longer beats that baseline on a clear majority of tasks, the claim of superior transferable representations is not supported.","tokens_in":17365,"feed_emoji":"⚛️","tokens_out":4830,"duration_ms":42395,"temperature":0.7,"pith_summary":"The paper sets out to show that a single transformer foundation model, pretrained on heterogeneous, multi-rate tokamak measurements, can be fine-tuned to outperform task-specific CNN models across a wide variety of reconstruction and forecasting objectives. It claims that on 13 of 14 tasks in the MAST benchmark, fine-tuned TokaMind achieves lower error than the provided CNN baseline, and that warm-starting from the pretrained checkpoint beats training the same architecture from scratch under a matched epoch budget, especially for long-horizon, high-frequency targets. If true, this would mean fusion researchers can share one reusable representation of plasma dynamics instead of training specialized models for each diagnostic and objective. The paper also shows that a much smaller version retains most of the benefit, and that a training-free DCT-based compression is competitive with a learned autoencoder embedding.","feed_headline":"Pretrained transformer tops 13 of 14 tokamak benchmark tasks","feed_subtitle":"Fine-tuning from broad MAST pretraining beats from-scratch training on the hardest plasma forecasting problems.","key_machinery":"The load-bearing mechanism is the tokenizer built on the DCT3D codec: any windowed chunk of a time-series, profile, or video is reshaped to a 3D tensor, transformed with an orthonormal 3D discrete cosine transform, and truncated to the low-frequency coefficients that retain a target fraction of the signal energy. This yields a fixed-size, training-free, energy-preserving token for every chunk. The token encoder then adds learned embeddings for signal identity, modality, role (input vs actuator), and recency, and the transformer backbone with masked self-attention pools the tokens into a [CLS] representation. The output decoder splits into modality-specific heads and per-target adapters, so t","core_discovery":"The central claim is that broad multi-signal pretraining yields transferable representations of tokamak plasma dynamics. TokaMind tokenizes windows of time-series, profile, and video diagnostics into fixed-size embeddings using a three-dimensional discrete cosine transform (DCT3D), then processes the variable-length token set with a masked transformer. After pretraining on a reconstruction objective over all available MAST signals, task-specific fine-tuning of the output adapters and select blocks improves test NRMSE over the benchmark CNN baseline on 13 of 14 tasks, with group-level gains between 7% and 20% relative to the baseline. Warm-start fine-tuning also outperforms training from scra","pith_inferences":["If the transferable representations generalize beyond MAST to other tokamaks, the same pretraining recipe could cut the data and compute needed for new devices, but the paper does not demonstrate cross-device transfer.","The two-stage freezing strategy (first freeze backbone, then freeze token encoder) suggests a cheap adaptation protocol that could be run on edge hardware during experiments, though real-time latency is not measured.","Because the benchmark and the baseline were created by a group overlapping with the model's authors, an independent re-evaluation on a third-party dataset would strengthen the claim that the gains are due to the foundation-model design rather than the evaluation setting.","The DCT3D tokenization is generic; the same interface could be applied to other multi-channel scientific time-series where signals have different rates and gaps."],"forward_implications":["A single pretrained checkpoint can be reused across many tasks by only swapping small output adapters, reducing the training cost per task.","Warm-starting from pretrained weights is especially valuable for long-horizon and high-frequency forecasting, where from-scratch training is unstable or underfits.","A model with roughly 5 million parameters (Tiny) captures most of the benefit, suggesting deployable models for real-time analysis.","The schema-flexible tokenization handles missing signals and changing input sets without imputation, making the approach practical for noisy experimental data.","The training-free DCT3D embedding is competitive with a learned VAE, so the framework does not depend on expensive pretrained codecs."],"fun_headline_variants":["Open-source tokamak model wins 13 of 14 benchmark tasks","TokaMind: pretrained transformer tops plasma forecast tests","Multi-modal transformer beats baseline on 13 of 14 fusion tasks","Warm-start plasma model excels on hardest forecasting challenges","New AI foundation model for tokamak sets benchmark record"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim that TokaMind's gains demonstrate transferable representations rests on the benchmark's CNN baseline being a strong and fairly designed comparison; if the baseline is underpowered or the tasks are implicitly shaped to the transformer design, the comparative results would not establish a general foundation model.","fun_headline_variants_meta":{"raw":{"variants":["Open-source tokamak model wins 13 of 14 benchmark tasks","TokaMind: pretrained transformer tops plasma forecast tests","Multi-modal transformer beats baseline on 13 of 14 fusion tasks","Warm-start plasma model excels on hardest forecasting challenges","New AI foundation model for tokamak sets benchmark record"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1097,"prompt_tokens":790,"completion_tokens":307,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":222}},"tokens_in":534,"tokens_out":307,"duration_ms":3427,"temperature":1.0,"reasoning_tokens":222,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:05:33.280194+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a CNN baseline with the same parameter count, same total training epochs, and the same input augmentation and dropout used for TokaMind, then re-run all 14 benchmark tasks. If the fine-tuned transformer no longer beats that baseline on a clear majority of tasks, the claim of superior transferable representations is not supported.","supporting_citations":[],"review_version":1}