{"id":"b0283ab6-4889-4fe4-b88a-a6aabdbc3b42","arxiv_id":"2412.13498","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An attention-augmented CNN identifies tearing-mode magnetic islands in EAST tokamak ECE signals with 91.96% test accuracy, a modest gain over a plain CNN.","lead":"A fusion experiment classifier uses an attention-aware convolutional neural network on electron cyclotron emission signals from the EAST tokamak to detect tearing-mode magnetic islands, reaching about 92 percent test accuracy. The work is one step toward real-time identification of a plasma instability that matters for ITER operations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 91.96% accuracy may reflect the model learning the ECE-based labeling rule rather than an independent tearing-mode ground truth; a blinded relabeling test with non-ECE diagnostics is needed.","rationale":"After a good-faith reading, the strongest evidence—the 91.96% accuracy and the 4.77% attention improvement—depends on the labels being correct and independent. The paper's own Section 2.1 ties labels to a correlation analysis between ECE and magnetic probe signals, while the model uses only ECE signals. That overlap is the least secure point in the argument: a sufficiently expressive CNN could learn the same ECE-based heuristic the labelers used, making the test-set accuracy an agreement score with the labeling procedure rather than a measure of physical detection. The reader identified this same assumption, and I agree. Other weaknesses (no error bars, no multiple seeds, metric inconsistencies) are secondary: the apparent metric discrepancies are explainable by different thresholds in Figure 6, and additional seeds would not correct a biased label source. The paper does deserve credit for a shot-level train/test split and for reporting a confusion matrix; those are proper evaluation steps, but they do not resolve the label-independence question. The proposed relabeling test would settle it, so the reader's conditional verdict should stand unchanged.","tokens_in":9806,"tokens_out":11230,"duration_ms":105300,"concrete_test":"Independent relabeling check: take the 3161 test-set windows from the 16 held-out discharges and have an expert (or a rule-based algorithm) assign labels using only Mirnov probe and SXR data, blinded to ECE signals and to model predictions. Recompute the AM-CNN confusion matrix against these ECE-blind labels. If accuracy drops by more than about 5 percentage points from 91.96%, the reported performance is inflated by ECE-derived label coupling; if accuracy is maintained within that band, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 states that labels are obtained from Mirnov probes, ECE, and SXR, and that the presence of a tearing mode is determined by performing a correlation analysis between the ECE signals and the magnetic probe signals, followed by manual curation. The model input is exclusively the same 11 ECE channels (Section 2.2.1, Figure 3). Because the ground-truth labels are partly derived from the same ECE features (phase inversions, temperature flattening) that the classifier consumes, the reported 91.96% accuracy may measure how well the model reproduces the labeling heuristic rather than how well it detects magnetic islands under an independent ground truth. This is the load-bearing premise of the central claim: if the labels are not independent of the input, the accuracy and the 4.77% attention gain do not establish real tearing-mode detectability. The paper provides no code, data, inter-labeler agreement, or diagnostic cross-check to rule this out. This is a circularity concern, not an allegation of scientific misconduct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an attention-aware convolutional neural network (AM-CNN) for binary classification of the presence of tearing-mode magnetic islands using 11 channels of electron cyclotron emission (ECE) from the EAST tokamak. A dataset of 68 discharges between 2016 and 2021 is split by shot into training, validation, and test sets. The authors report a test accuracy of 91.96%, with improvements over a plain CNN in accuracy, recall, and F1. The paper also shows a time-evolution analysis of one test discharge and connects model behavior to the physics of island formation.","tokens_in":9991,"tokens_out":8119,"duration_ms":60610,"significance":"If the central claim holds, the work is a relevant step toward real-time tearing-mode identification on EAST and could inform future ITER control. The paper has notable strengths: a shot-level test split that prevents the most obvious temporal leakage, a physically motivated attention mechanism, and a direct comparison with a plain CNN. The main weaknesses are the internal inconsistency of the reported metrics, the partial dependence of the labels on the same ECE diagnostic used as input, and the absence of uncertainty estimates. These issues currently prevent full confidence in the reported accuracy and in the claimed attention gain.","major_comments":[{"comment":"The reported performance numbers are internally inconsistent. The abstract states a classification accuracy of 91.96%; Section 3.1 states an accuracy of 92.41% and F1-score of 0.9442, and also reports TPR 92.99% and FPR 8.9% at threshold 0.2170. The confusion matrix in Figure 6(a) gives TN=900, FP=78, FN=176, TP=2007, which implies accuracy = (900+2007)/3161 = 91.96%, TPR = 2007/2183 = 91.94%, FPR = 78/978 = 7.98%, and F1 ≈ 0.940. These values do not match the text in Section 3.1. The manuscript needs a single consistent set of evaluation numbers, with the operating threshold and class prior specified, so that the central accuracy claim is unambiguous.","section":"Abstract; Section 3.1; Figure 6"},{"comment":"The label construction creates a potential circularity. The labels are described as obtained from Mirnov probes, ECE, and SXR, and as determined by correlation analysis between ECE signals and magnetic probe signals (Section 2.1). The classifier input is exclusively the same 11 ECE channels (Section 2.2.1). If the labeling rule relies on the same ECE features (phase inversions, temperature flattening) that the model consumes, the reported 91.96% accuracy may partly measure the model's ability to reproduce the labeling heuristic rather than its ability to detect tearing modes under an independent ground truth. This is a load-bearing premise for the central claim. A blinded relabeling test using only non-ECE diagnostics (e.g., Mirnov and SXR), or at minimum a quantitative agreement analysis between ECE-derived and non-ECE-derived labels, is needed.","section":"Section 2.1"},{"comment":"The comparison between AM-CNN and CNN reports point estimates only, with no repeated runs or confidence intervals. The text in Section 3.2 saying 'The AM-CNN model is trained 100 times' appears to refer to 100 training epochs, not to 100 independent training runs. Without multiple seeds or bootstrap confidence intervals, the claimed improvements (accuracy +4.77%, recall +5.5%, F1 +3.74%) are not shown to be statistically significant, and the reader cannot assess the stability of the 91.96% test accuracy.","section":"Section 3.1; Figure 5"},{"comment":"The description of the attention mechanism is not sufficiently detailed to reproduce the method. Equation (1) is the standard dot-product attention formula, but the text states that 'a 2×1 matrix is set as keys' and that the weights are dot-multiplied with the input, without defining the dimensions of Q, K, V, how the 26×11×20002 input is mapped to queries/keys/values, or where exactly the attention is inserted relative to the convolutional and pooling layers. Since the attention module is the central novelty and the claimed gain over plain CNN rests on it, the architecture description needs to be precise enough for an independent implementation.","section":"Section 2.2.2; Equation (1)"}],"minor_comments":[{"comment":"The dataset size is described inconsistently: the abstract says '2.5 × 10^9 data', Section 1 says '2.5 × 10^9 samples', and Section 2.1 says '11362×220022 samples'. Since 11362 × 220022 ≈ 2.5 × 10^9, the latter is the total number of points across all samples; the terminology should be made uniform and the number of independent samples (11362) should be stated clearly.","section":"Abstract; Section 2.1"},{"comment":"The chronological ordering of the data split should be stated explicitly: the 16 test shots (61042–86559) are earlier than the training shots (91364–103486). If this is an intentional chronological split, the authors should say so and discuss the resulting domain shift; if not, the split order should be justified.","section":"Section 2.1; Figure 1"},{"comment":"The denoising process applied to the ECE signals is mentioned but not described. A brief description of the denoising method is needed for reproducibility.","section":"Section 2.1"},{"comment":"The attention formula in Equation (1) omits the 1/sqrt(d_k) scaling that is standard in the dot-product attention of the cited Transformer paper [28]. The authors should state whether the scaling is used.","section":"Equation (1)"},{"comment":"The definition of precision in the text ('Precision measures the proportion of instances in which the model predicts positive samples to be positive') is incorrect; precision is the proportion of predicted positives that are true positives. The phrasing should be corrected.","section":"Section 3.1"},{"comment":"The percentile histogram in Figure 2(c) is not defined; the text should state whether the percentiles are computed over shots, over time samples, or over all data points.","section":"Figure 2(c)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal as an applied deep-learning study for fusion diagnostics. In my reading, the central claim is plausible but not yet fully evidenced due to the inconsistent evaluation numbers and the label-independence question. These are fixable with additional analysis, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate, modest incremental result — an attention-augmented CNN for classifying tearing-mode vs non-tearing-mode windows from EAST ECE data. The headline number (91.96% accuracy on a held-out shot-level test set) is real in the sense that it is reproduced by the confusion matrix in Figure 6. The paper is above the median for this genre because the split is by discharge, not by random sample, and the baseline comparison uses the same train and test sets. It is not a breakthrough; the components are standard and the novelty is the application, not the architecture.\n\nWhat it does well: shot-level split, explicit class balance (30.94% negative / 69.06% positive), denoising and time-slice details, and an honest acknowledgment that early-stage / small-island detection is difficult. The attention mechanism is applied at a sensible place (after the fully connected layer) and the authors give a physical rationale for why attention should help — focusing on phase-inversion and temperature-flattening time points.\n\nThe soft spots are real but not fatal. First, internal inconsistencies: the confusion matrix implies TPR=91.94%, FPR=7.98%, F1≈0.9405, while the text reports TPR 92.99%, FPR 8.9%, F1 0.9442, and accuracy 92.41% elsewhere. These need reconciliation. Second, no code or data is released, and there are no multiple-seed runs or confidence intervals, so we cannot tell how stable the 4.77% attention gain is. Third, and most important, the label construction is under-specified. Section 2.1 says labels come from Mirnov, ECE, and SXR and are set by correlating ECE with Mirnov signals. Since the model's input is exclusively ECE, the ground truth is not fully independent of the feature space. This is a classic circularity concern, but it is not damning: if Mirnov and SXR provided the primary label and ECE was only used to confirm timing, then the task is legitimate. The paper needs to say which diagnostic anchored the label and ideally provide a blinded relabeling check or release enough data for an external audit.\n\nOne more thing: the test set (shots 61042–86559) is earlier in time than the training set (91364–103486). That is a harder and more honest generalization test, but the authors should state it explicitly; as written it looks like an arbitrary split.\n\nWho is this for? Plasma-control and ML-for-tokamak researchers who want a baseline classifier for ECE-based island detection. A serious referee should engage with it. My recommendation: send to peer review, and require the consistency fixes, a label-independence statement, and ideally code/data release. The core result is plausible; it just needs transparency.","headline":"A legitimate, modest classifier paper with a real shot-level test split and a plausible central number; main issues are transparency (code/data, label independence) and internal metric inconsistencies, not the basic approach.","tokens_in":10569,"tokens_out":3632,"would_cite":false,"duration_ms":34343,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An attention-aware convolutional neural network can identify tearing-mode magnetic islands from electron-cyclotron-emission signals with 91.96% accuracy, outperforming a plain CNN by 4.77 points in accuracy, 5.5 in recall, and 3.74 in…","keywords":["tearing mode","magnetic island","electron cyclotron emission","convolutional neural network","attention mechanism","tokamak","deep learning","EAST"],"falsifier":"Re-label a held-out set of the same 68 discharges using only Mirnov-probe and soft-X-ray signals, with no ECE input, then compare the AM-CNN predictions to those labels; if accuracy on the ECE-free labels is much lower than 91.96%, the original accuracy was inflated by ECE-based label leakage.","tokens_in":9626,"feed_emoji":"🧲","tokens_out":6097,"duration_ms":51747,"temperature":0.7,"pith_summary":"This paper tries to show that an attention-aware convolutional network can reliably identify tearing-mode magnetic islands from electron-cyclotron-emission (ECE) temperature signals on a tokamak. The authors build a classifier from 68 EAST discharges and report 91.96% accuracy on a held-out set of discharges, with the attention module improving accuracy by 4.77 percentage points, recall by 5.5 points, and F1 by 3.74 points over a plain CNN. If the result holds, it would give fusion control systems a fast, automated way to spot islands as they form, a key step toward preventing disruptions in future reactors. The main caveat is that the same ECE data both feeds the model and helped create the labels, so the reported accuracy depends on those labels being correct and independent.","feed_headline":"Attention CNN spots magnetic islands in tokamak data at 92 percent","feed_subtitle":"Attention module lifts plain-CNN accuracy by 4.8 points and recall by 5.5 on EAST electron-cyclotron signals.","key_machinery":"The central mechanism is a dot-product attention module attached to the CNN's final feature layer. In this module, a learned query multiplies a key matrix, and a softmax produces attention weights used to weight and sum the value representations at each time point; mathematically it computes $\\text{Attention}(Q,K,V)=\\operatorname{softmax}(QK^{T})V$. The attention focuses the classifier on the ECE time samples where the flattening and phase reversal of electron-temperature signals mark a rotating magnetic island, which is why the authors say the model gains accuracy and recall over a plain CNN.","core_discovery":"The central claim is that a CNN augmented with a dot-product attention module (AM-CNN) can detect the presence of tearing-mode magnetic islands from 11 ECE channels spanning the plasma core to edge, reaching 91.96% classification accuracy on a test set of 16 EAST discharges. The improvement over a CNN without attention is quantified as +4.77% accuracy, +5.5% recall, and +3.74% F1-score; at a 0.2170 threshold the model also achieves a true-positive rate of 92.99% and a false-positive rate of 8.9%. The authors attribute the gain to the attention mechanism weighting the time points where the rotating island creates phase-reversed temperature fluctuations across channels, and they note that detection is most reliable when the island is mature and less reliable during seed-island onset.","pith_inferences":["Because ECE feeds both the model and the manual labels, the reported test accuracy may overstate real-world performance; a label source independent of ECE would give a more honest estimate.","The model's weaker performance during island onset suggests that labeling early-stage, small islands as positive is difficult; retraining with finer onset labels could improve seed-island detection.","The architecture is general enough that the same attention-plus-CNN setup could likely be applied to locked-mode detection or to other diagnostics, though the paper does not demonstrate this."],"forward_implications":["An AM-CNN trained on 11 ECE channels can classify tearing-mode presence in held-out EAST discharges with 91.96% accuracy, supporting real-time island identification.","Adding the dot-product attention module improves accuracy by 4.77 percentage points, recall by 5.5 points, and F1 by 3.74 points over a plain CNN on the same train/test split.","At a decision threshold of 0.2170, the model reaches a 92.99% true-positive rate and an 8.9% false-positive rate on the test set.","The model is most reliable once the magnetic island is mature and the temperature reversal surface is distinct; early-stage and seed-island detection remains less stable.","The authors propose extending the work to locked-mode detection, island width and position estimation, and additional diagnostics such as magnetic probes and soft X-ray systems."],"supporting_citations":[{"why":"Supplies the dot-product attention mechanism that the AM-CNN builds on.","marker":"[28]"},{"why":"Provides the standard convolutional neural network architecture and context that the model extends.","marker":"[27]"},{"why":"Establishes a prior deep-learning approach for MHD mode identification on EAST that this work builds toward.","marker":"[21]"},{"why":"Motivates ECE-based tearing-mode detection and control evaluation for ITER.","marker":"[17]"},{"why":"Describes the EAST ECE diagnostic upgrade and establishes the signal sensitivity the model relies on.","marker":"[24]"},{"why":"Documents the ECE temporal and radial resolution that lets the data capture magnetic-island fluctuations.","marker":"[25]"},{"why":"Provides a machine-learning control baseline for disruption and tearing-mode avoidance that motivates the real-time goal.","marker":"[20]"}],"fun_headline_variants":["Attention CNN detects magnetic islands in EAST tokamak at 92%","AM-CNN identifies magnetic islands in tearing mode on EAST with 92% accuracy","Attention boosts CNN for magnetic island detection in tokamak to 92%","92% accuracy: attention CNN spots magnetic islands on EAST","Attention-aware CNN finds magnetic islands in EAST tearing mode at 92%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth labels for tearing-mode presence are accurate and independent of the ECE features the model sees, since ECE signals were part of the labeling process and any label errors or leakage could inflate the reported 91.96% accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Attention CNN detects magnetic islands in EAST tokamak at 92%","AM-CNN identifies magnetic islands in tearing mode on EAST with 92% accuracy","Attention boosts CNN for magnetic island detection in tokamak to 92%","92% accuracy: attention CNN spots magnetic islands on EAST","Attention-aware CNN finds magnetic islands in EAST tearing mode at 92%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000395,"raw_usage":{"total_tokens":2128,"prompt_tokens":1057,"completion_tokens":1071,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":975}},"tokens_in":673,"tokens_out":1071,"duration_ms":7665,"temperature":1.0,"reasoning_tokens":975,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:03:39.037457+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label a held-out set of the same 68 discharges using only Mirnov-probe and soft-X-ray signals, with no ECE input, then compare the AM-CNN predictions to those labels; if accuracy on the ECE-free labels is much lower than 91.96%, the original accuracy was inflated by ECE-based label leakage.","supporting_citations":[],"review_version":1}