{"id":"45420f6f-4484-4433-87d2-a5cd0b3101b7","arxiv_id":"2607.15719","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A flare-prediction CNN focuses on magnetic regions whose extracted parameters predict flares as well as standard physics-based masks, and these regions show a single-polarity-dominant complexity.","lead":"Using an explainable-AI method called Grad-CAM, the paper finds which parts of solar magnetograms a flare-prediction neural network actually focuses on, then extracts magnetic parameters from those regions to test whether they carry real flare information. The result is a quantitative check that the model's attention regions are physically meaningful: they predict flares about as well as standard physics-based active-region masks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control for arbitrary MFR masks: comparable predictive power could be a byproduct of mask size, not Grad-CAM's selectivity","rationale":"The reader correctly identified the faithfulness of Grad-CAM and the OTSU thresholding as the weakest assumption. The most load-bearing concern is that the paper provides no null control to rule out that the MFR selection is essentially arbitrary. The comparison to the SHARP mask is not sufficient, because SHARP is the whole active region; a random subregion of comparable size would likely yield similar parameters. This missing baseline undermines both the predictive-power argument and the PII-complexity argument, since both derive from the Grad-CAM-derived masks. The proposed random-mask test directly addresses this by generating matched null masks and testing whether the real MFRs add predictive information beyond size/location alone. The reader's verdict of CONDITIONAL remains appropriate: the study is methodologically solid in many respects (clean data pipeline, two-stage split, multiple classifiers), but the central physical-meaningfulness claim currently rests on an unvalidated assumption. No additional issues change the verdict; the condition is exactly the need for this null control. Therefore, the verdict should stay UNCHANGED, with the existing conditional acceptance relying on the authors providing such a control in revision.","tokens_in":15087,"tokens_out":4497,"duration_ms":41341,"concrete_test":"For each sample in the test set, generate 100 random binary masks within the SHARP boundary that exactly match the size distribution, connected-component structure, and (for the PII analysis) the sign-split of the true MFR masks. Compute the same 16 magnetic parameters on each random mask and retrain the same three classifiers (RF, FCNN, SVM) with the same hyperparameter grid, then compare the resulting TSS/F1 distributions against the real MFR-mask results in Table 6. Additionally, compute the PII distributions from random masks using the same thresholding and area-weighting recipe and compare to Figure 6. If random masks achieve statistically indistinguishable predictive performance and PII separation, the central claim fails; if real MFRs significantly outperform random masks, the physical meaningfulness is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MFRs are physically meaningful rests on two observations: (1) magnetic parameters extracted from MFRs have predictive power comparable to those from the SHARP mask (Tables 6–7), and (2) the PII distribution in positive vs. negative activation areas appears to distinguish flaring from non-flaring regions (Figure 6). Neither observation is tested against a null hypothesis. The SHARP mask covers the entire active region; any moderately sized interior region will contain most of the magnetic flux and therefore yield similar parameter values. Without a random-region baseline matched in size and location to the MFRs, the 'comparable predictive power' result only shows that the MFR mask contains a substantial fraction of the active region, not that the specific regions highlighted by Grad-CAM are physically special. Similarly, the PII analysis assigns polarity imbalance to positive and negative activation areas, but the sign of the heatmap is not validated to correspond to excitatory/inhibitory effects; OTSU thresholding of the absolute value plus area-weighted averaging over connected components introduces arbitrary choices. The paper's own Discussion acknowledges limitations such as pixel scale and data leakage but does not mention the absence of a null control for the MFR selection itself. This is the load-bearing weakness because if the MFR masks were replaced with random subsets, the same quantitative results might emerge, collapsing the claim that CNNs learn physically meaningful representations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a quantitative XAI framework for CNN-based solar flare prediction. A ResNet-style CNN is trained on SDO/HMI SHARP vector magnetograms from 2699 HARPs with a two-stage, active-region-aware split. Grad-CAM (with both positive and negative activations retained) is used to define model-focused regions (MFRs) via OTSU thresholding of the absolute heatmap. The authors extract 16 standard SHARP magnetic parameters from MFR, SHARP, and PIL masks, and compare their predictive power in SVM, RF, and FCNN classifiers. They also introduce a Polarity Imbalance Index (PII) computed from LOS magnetogram pixels inside positive and negative activation areas, reporting that flare-productive regions have intermediate polarity imbalance in positive activation areas (PII_p ~ 0.33) while negative activation areas are more unipolar (PII_n ~ 1). The paper concludes that CNNs learn physically meaningful magnetic-complexity representations.","tokens_in":15412,"tokens_out":4005,"duration_ms":38276,"significance":"If the claims hold, the paper would demonstrate that XAI can move beyond qualitative heatmaps to quantitative physical validation, a valuable step for both solar physics and trustworthy AI. The authors are careful in several respects: they use active-region-based splitting to mitigate leakage, compare against the established SHARP parameter benchmark, use multiple heterogeneous classifiers with hyperparameter search, and make code and data publicly available. The PII construction is simple and directly tied to the CNN's attention. However, the central inference—that the specific regions highlighted by Grad-CAM are physically special—is not tested against a null model, and the PII analysis lacks uncertainty quantification. These are load-bearing gaps, because the paper's main conclusions rest on the selectivity of the MFR masks and on the statistical reality of the PII distribution differences.","major_comments":[{"comment":"The claim that MFR-derived parameters are 'comparable' to SHARP-derived parameters is not supported by uncertainty quantification. Table 6 reports mean metrics over hyperparameter configurations but no standard deviations, confidence intervals, or significance tests. Many differences are tiny (e.g., RF TSS: MFR 0.5408 vs SHARP 0.5543; FCNN F1: MFR 0.9067 vs SHARP 0.9079). Without paired tests across random seeds and configurations, 'comparable' is not established beyond visual inspection. Please report variability and perform significance tests on the metric differences.","section":"§4.1, Tables 6–7"},{"comment":"There is no null control for the MFR mask itself. The SHARP mask covers the whole active region; any reasonably large, centrally located region will contain most of the magnetic flux and may yield similar parameter values. To show that Grad-CAM's selectivity matters, the parameter comparison should include random masks matched to the MFR masks in size, location, and contiguity (or a permutation/randomization of the Grad-CAM heatmap). Without such a control, the 'comparable predictive power' result is compatible with the MFR mask being a non-selective subsample, which would undermine the paper's central claim that the CNN has learned physically meaningful spatial features.","section":"§4.1, §2.2"},{"comment":"The PII analysis is currently qualitative and based on several unvalidated choices. The paper reports KDE peaks (PII_p ~ 0.33, PII_n ~ 1) but provides no significance tests, confidence intervals, or effect sizes for the differences between positive and negative events. The definition of PII depends on an ad hoc ±150 G threshold, on OTSU thresholding of the absolute heatmap, on the sign convention of Grad-CAM activations, and on area-weighted averaging over connected components; none of these choices is tested for robustness. The sign of the Grad-CAM activation is assumed to indicate flare-promoting vs. suppressing regions without independent validation. Please add sensitivity analyses and formal statistical comparisons, and ideally validate the sign interpretation with an alternative attribution method or ablation.","section":"§4.2, Eq. (3), Fig. 6"},{"comment":"The analysis of MFR predictive power is partly circular: the Grad-CAM masks are produced by a CNN trained on the same flare labels used to evaluate the MFR-derived parameters. The SHARP-mask comparison provides a useful external anchor, but it does not eliminate the possibility that the MFR parameters inherit predictive power from label-informed region selection. I recommend an additional control, e.g., evaluating parameters from Grad-CAM masks of an untrained or label-free CNN, or showing that MFR masks from a model trained on a different task do not yield comparable predictive power.","section":"§4.1, §2.1"}],"minor_comments":[{"comment":"Line breaks in the title ('Model-F ocused') and several inline artifacts ('T able', 'V F ully', 'demostrate') should be fixed. The abstract's phrase 'magnetically complex configurations that are dominated by a single polarity' is slightly confusing given the later 'intermediately imbalanced' characterization; consider rewording for clarity.","section":"Title/Abstract"},{"comment":"Duplicate entries appear: Breiman (2001) twice, Schrijver (2007a/b) twice, and Li et al. (2025a) and (2025b) appear to be the same reference. Please consolidate.","section":"References"},{"comment":"The statement that the dataset comprises 'around half of all HARPs' is vague. Please give the exact selection criteria and the number of excluded HARPs due to central-meridian, NOAA-association, or NaN filtering.","section":"§2, Data"},{"comment":"Table 5 lists 'random seed' as a hyperparameter, but the text says '60 different hyperparameter configurations' without explaining how the grid yields 60. Please clarify the counting.","section":"§3.2, Hyperparameters"},{"comment":"The red/blue color coding in Figure 6 is described only in the caption; for accessibility, consider adding explicit labels or a colorblind-safe palette. Also, the caption says 'different solar active region events'—please specify whether each point is one timestamp or an aggregate over the active region's lifetime.","section":"§4.2, PII"},{"comment":"The discussion of PIL mask limitations is useful, but it would be strengthened by reporting the number of samples excluded in the 3-group vs 2-group alignments (i.e., how many HARPs lack PIL masks). Table 7 already shows the counts indirectly; please state them explicitly.","section":"§5, Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important question, and the authors have made a genuine effort to be transparent about data and limitations. The central scientific claim, however, rests on the selectivity of Grad-CAM-derived masks and on the statistical reality of the PII separation. Both need additional controls: random-mask baselines, uncertainty quantification, and sensitivity analyses. These are feasible within the scope of the manuscript and do not require new physical theory. I therefore support a major revision rather than rejection. I would also flag that the 'comparable predictive power' language should be calibrated with significance tests; otherwise reviewers in the broader ML community may view the evidence as anecdotal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fairly good paper, with one gap that keeps it from fully landing. The genuinely new thing is quantitative: instead of just showing Grad-CAM heatmaps pile up near PILs, the authors extract 16 magnetic parameters from the model-focused regions, compare them against SHARP and PIL masks with three classifiers, and add a polarity-imbalance index (PII) that tries to characterize what the CNN attends to. That PII result — flare-productive ARs show intermediate polarity imbalance, one dominant polarity rather than balanced or unipolar — is a concrete, falsifiable statement about what the CNN has learned, and it is consistent with known solar physics. They also handle data splitting carefully (two-stage split to expose active-region leakage), and they are transparent about limitations: scale changes, leakage, misidentified ARs. Code and data are public. Credit where due.\n\nThe soft spot is the one the stress-test flags: there is no null control for the MFR mask itself. The main evidence that MFRs are physically special is that parameters extracted from them predict flares as well as SHARP parameters. But SHARP masks cover the whole active region, and any reasonably sized interior region will contain most of the flux and yield similar parameters. Without a random-region baseline matched in size and location, the \"comparable predictive power\" result only shows the MFR contains a decent chunk of the AR, not that Grad-CAM picked the physically relevant part. That is load-bearing for the paper's central claim. The PII analysis partially escapes this criticism because it compares positive vs negative activation areas within MFRs, and the difference (peak PII ~0.33 vs ~1) is striking. But it inherits the same unvalidated Grad-CAM semantics: the sign of the heatmap is assumed to mean excitatory/inhibitory, and no attribution sanity check (e.g., another method, or ablation) is offered.\n\nMinor but real: key numbers lack error bars and significance tests; the PII threshold (±150 G) is hand-chosen; the 1:2 density-peak ratio is described as \"approximately\" with no uncertainty. The PIL underperformance is partly a mask-extent artifact, which the authors acknowledge.\n\nVerdict: serious referee, yes. The paper is well above the desk-reject line; it is a competent empirical study that will push the subfield toward quantitative validation of XAI. But the central inference needs the random-mask control and either error bars or a significance test on the PII comparison. I'd send it to review with a request for that baseline.","headline":"A competent quantitative XAI study whose central claim needs a random-mask baseline before it fully lands.","tokens_in":15883,"tokens_out":2325,"would_cite":true,"duration_ms":19639,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Flare-prone active regions favor one magnetic polarity, AI maps show","keywords":["solar flare prediction","convolutional neural networks","explainable AI","Grad-CAM","active regions","magnetic polarity imbalance","magnetograms","space weather"],"falsifier":"Ablation test: remove or zero the MFR mask pixels from input magnetograms and re-evaluate the CNN; if test-set performance does not degrade substantially relative to removing equal-area random regions, the MFRs are not the regions driving predictions and the physical interpretation collapses. A complementary check compares Grad-CAM masks with a different attribution method and requires high overlap.","tokens_in":14989,"feed_emoji":"☀️","tokens_out":6569,"duration_ms":55562,"temperature":0.7,"pith_summary":"This paper tries to show that a convolutional neural network trained to predict solar flares from magnetograms is not just a statistical black box: the regions it visually focuses on, isolated with the Grad-CAM attribution method, contain the same flare-relevant magnetic information that solar physicists extract from established active-region masks. The authors extract sixteen magnetic parameters from these model-focused regions (MFRs) and feed them to three standard classifiers, finding performance comparable to the standard full-region mask and better than a mask restricted to polarity inversion lines. They then define a polarity imbalance index and find that flare-producing active regions show an intermediate, single-polarity-dominated imbalance (roughly 1:2 positive-to-negative pixels), while quiet regions are more unipolar. The upshot is that deep-learning flare forecasts can be read as physical statements about magnetic complexity, making XAI a validation and discovery tool for solar physics, not just an explainability aid.","feed_headline":"Flare-prone active regions favor one magnetic polarity, AI maps show","feed_subtitle":"Attention regions singled out by the network carry as much flare signal as established physics-based masks.","key_machinery":"Two constructions carry the argument. Model-focused regions (MFRs) are binary masks formed by applying OTSU thresholding to the absolute value of the Grad-CAM heatmap, where Grad-CAM weights are the global-average-pooled gradients of the flare score with respect to the last convolutional layer's feature maps; unlike standard Grad-CAM, the heatmap keeps both positive and negative activations, so the mask contains regions that promote or suppress flare predictions. The Polarity Imbalance Index (PII) is defined as |Np − Nn|/(Np + Nn) using pixels with |Blos| > 150 G inside each activation region, aggregated by area-weighting across connected components. MFRs localize what the network uses; PII","core_discovery":"On the paper's own terms, the central discovery is that the regions a CNN attends to when predicting flares are physically meaningful. Using Grad-CAM on the last convolutional layer and OTSU thresholding of the absolute heatmap to define model-focused regions (MFRs), the authors compute sixteen standard magnetic parameters inside those regions. Across three classifiers (random forest, SVM, fully connected network), MFR-derived parameters predict ≥C-class flares within 24 hours about as well as parameters from the full SHARP active-region mask, and clearly better than parameters from polarity-inversion-line masks. The second finding is a polarity signature: within positive-activation areas of","pith_inferences":["One natural extension is to test PII as a standalone, line-of-sight-only flare predictor; if the distributional separation seen here holds, a simple pixel-count ratio could offer a lightweight operational complement to full network forecasts.","The roughly 1:2 dominant-polarity mixture may be a general property of flare-productive regions; checking it against independent active-region catalogs across multiple solar cycles would tell whether it is a stable physical signature or particular to this dataset.","A direct ablation — masking out MFR pixels and measuring how much CNN performance drops relative to random masks — would distinguish regions the network genuinely relies on from correlated bystanders; the paper does not report this test.","The difference between the leakage-free and leaky test splits (TSS 0.734 vs 0.762) implies that operational use on new active regions may underperform benchmark numbers, so chronological, region-disjoint evaluation deserves to become the default in this field."],"forward_implications":["MFR-derived magnetic parameters predict flares as well as parameters from the full active-region mask, so attention-based localization preserves the flare-relevant information in the magnetogram.","Polarity-inversion-line masks underperform because their limited spatial extent discards too much magnetic context; prediction systems should avoid relying on PIL strips alone.","Flare-productive active regions are characterized by intermediate polarity imbalance with one polarity dominating, giving a simple, observable discriminator for flare forecasting.","Grad-CAM's positive and negative activation regions separate cleanly by magnetic complexity, so CNN heatmaps can be decomposed into physical components rather than treated as opaque saliency.","If these results hold, CNN flare predictors can double as physical-discovery instruments, pointing to the specific magnetic configurations that precede flaring."],"fun_headline_variants":["AI's flare focus areas hold as much signal as physics masks","CNN attention spots predict flares like traditional masks do","Flare-prone zones show polarity imbalance, AI maps confirm","Explainable AI pinpoints single-polarity flare hotspots","Magnetic complexity in AI-focused regions foreshadows flares"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire physical interpretation rests on Grad-CAM heatmaps faithfully marking the image regions the CNN actually uses; if the heatmaps highlight dataset shortcuts instead, the MFR parameters and polarity-imbalance results describe the model, not the Sun.","fun_headline_variants_meta":{"raw":{"variants":["AI's flare focus areas hold as much signal as physics masks","CNN attention spots predict flares like traditional masks do","Flare-prone zones show polarity imbalance, AI maps confirm","Explainable AI pinpoints single-polarity flare hotspots","Magnetic complexity in AI-focused regions foreshadows flares"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3039,"prompt_tokens":794,"completion_tokens":2245,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":2166}},"tokens_in":538,"tokens_out":2245,"duration_ms":13735,"temperature":1.0,"reasoning_tokens":2166,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T22:29:54.244737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ablation test: remove or zero the MFR mask pixels from input magnetograms and re-evaluate the CNN; if test-set performance does not degrade substantially relative to removing equal-area random regions, the MFRs are not the regions driving predictions and the physical interpretation collapses. A complementary check compares Grad-CAM masks with a different attribution method and requires high overlap.","supporting_citations":[],"review_version":1}