{"id":"0c2b2ac6-f009-4367-9da1-417b52c23f2b","arxiv_id":"2509.05768","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 10-year, 39-country, 20-energy-type European electricity benchmark shows existing forecasting models, especially transformers, degrade under time-varying cross-energy correlations.","lead":"This paper introduces Real-E, a large European electricity generation dataset built from ENTSO-E data, and benchmarks over 20 forecasting models on it. It reports that graph-based models generalize better than transformer-based models when correlations between energy sources shift over time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark conclusions rest solely on Germany/France subset; no evidence these two countries represent Real-E's 39-country complexity, so the central generalization claim is unsupported.","rationale":"The reader's weakest_assumption is exactly that the Germany/France subset is a valid representative proxy for the entire dataset. My analysis independently converges on this as the most load-bearing concern: the central scientific claim—about model generalization on Real-E—is built entirely on empirical results from two countries. If those two countries are not representative, the paper's main conclusion collapses to a narrow case study, which would not support the title's promise of a 'Foundation Benchmark' for robust and generalizable forecasting. The TGV/GSD metric comparison is also confounded by dimension and resolution, but that is secondary to the representativeness issue. I agree with the reader's identification; my concrete test is designed to settle the concern. A conditional verdict is appropriate: the paper could be acceptable if the authors either demonstrate representativeness (e.g., by running the benchmark on additional countries) or revise the claims to be explicitly scoped to Germany and France. Given the existing internal inconsistencies in the reported numbers, the verification burden is high, but the representativeness concern is the most consequential for the central claim.","tokens_in":7460,"tokens_out":9896,"duration_ms":106939,"concrete_test":"Run the identical benchmark protocol (same data splits, hyperparameter grid, evaluation metrics, and seeds) on at least three additional countries from Real-E with different energy mixes—e.g., Spain, Italy, and Poland. Compare the ranking of model families (Spatial GNN vs Transformer vs Spectral GNN) and the TGV/GSD values against the Germany/France results. If the top-performing family changes or the transformer/spectral degradation is not consistently reproduced, the generalization claim to all 39 countries is unsupported; if results are consistent, the representativeness concern is mitigated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim—that existing forecasting models, especially transformer-based and spectral GNNs, generalize poorly on Real-E due to rapidly shifting cross-energy correlation structures—is empirically derived entirely from a two-country subset. Section 4 ('Dataset & Metrics') states: 'We choose a subset of proposed datasets, Real-E(Germany,France)...' and Section 3 limits the data analysis to 'Germany and France as representative cases.' However, the abstract and Section 1 claim Real-E covers 39 countries and 20 energy types, and the conclusions in Section 5 extrapolate to Real-E as a whole. No evidence is provided that Germany and France are representative of the full dataset's diversity (e.g., different energy mixes, regulatory regimes, data quality, or correlation dynamics). The TGV/GSD metric comparisons in Section 3 are also confounded by differing numbers of time series and temporal resolutions—TGV is a sum of Frobenius norms over correlation matrices and scales with matrix dimension, so cross-dataset comparisons without normalization are questionable. Furthermore, the reported degradation figures contain internal inconsistencies: the 85.4% transformer MAE increase and the 16.20% spectral GNN degradation do not match recomputations from Table 2 (e.g., transformer average on Electricity+Solar is ~0.117, on Germany+France ~0.215, an ~83.8% increase, not 85.4%; spectral GNN degradation is far larger than 16.2%). These issues make the empirical support for the central claim unreliable, but the most load-bearing gap is the unjustified leap from two countries to a 39-country generalization claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Real-E, a multivariate electricity forecasting dataset assembled from ENTSO-E, claimed to be the largest such dataset to date, with 39 European countries, 20 generation types, up to 9.5 years of history, and multiple temporal resolutions. The authors propose two metrics, Temporal Graph Volatility (TGV) and Graph Spectral Divergence (GSD), to measure time-varying cross-energy correlation structure, and report that Real-E has higher volatility than existing benchmarks. They benchmark 20 forecasting models from several architectural families on a two-country subset, Real-E(Germany, France), and conclude that Transformer-based and spectral GNN methods degrade sharply compared with their performance on Electricity and Solar benchmarks, while spatial GNNs generalize best. The dataset and benchmark code are promised via external links.","tokens_in":7714,"tokens_out":5341,"duration_ms":54227,"significance":"If the empirical claims are supported, Real-E would be a valuable resource for the time-series forecasting community: multi-country, multi-energy, long-horizon data with associated metadata is genuinely missing from current public benchmarks. The paper's systematic comparison of 20 baselines across architectural families and the introduction of correlation-shift metrics are useful contributions. The public-domain licensing and the explicit research questions are also strengths. However, the central generalization claims currently rest entirely on a two-country subset, the proposed metrics are not shown to be comparable across datasets of different dimensions, and some reported performance-degradation numbers are inconsistent with the paper's own table. These issues must be resolved before the paper's conclusions can be accepted.","major_comments":[{"comment":"The paper's cross-dataset generalization conclusions are drawn entirely from the Real-E(Germany,France) subset. Section 3 states 'we focus on Germany and France as representative cases' and Section 4 states 'We choose a subset of proposed datasets, Real-E(Germany,France).' No evidence is provided that Germany and France are representative of the full 39-country, 20-energy-type dataset in terms of energy mix, correlation dynamics, regulatory setting, or data quality. The abstract and Section 5 nevertheless generalize to Real-E as a whole. Because RQ1 and RQ2 are framed as Real-E-wide questions, this is a load-bearing gap. Please either benchmark on the full dataset or provide a representativeness analysis (e.g., TGV/GSD distributions across countries, energy-mix diversity, missing-data patterns).","section":"§3, §4, §5"},{"comment":"TGV and GSD are not normalized for matrix dimension. TGV in Eq. (1) is a sum of Frobenius norms of adjacency matrices of size m×n; GSD in Eq. (2) is an Euclidean distance between Laplacian eigenvalue vectors. If Germany/France use a different number of time series than Electricity or Solar, the higher TGV/GSD values for Real-E may reflect dimension scaling rather than genuinely more complex correlation dynamics. The text reports averages 'on existing benchmarks' and 'our dataset attains higher values' without stating the number of series in each dataset or any normalization. This weakens the central data-analysis claim. Please report normalized metrics or dimension-matched comparisons.","section":"§3, Metrics in Practice, Eqs. (1)-(2)"},{"comment":"The reported degradation rates are internally inconsistent with Table 2. For the four Transformer-based models, using the MAE values in Table 2 gives approximately (0.103+0.097+0.1105+0.156)/4 = 0.1166 on Electricity+Solar and (0.1845+0.2455+0.210+0.219)/4 = 0.2148 on Germany+France, an increase of about 84%, not 85.4%. For spectral GNNs, the category average MAE rises from about 0.1145 to 0.2235, an increase of about 95%, not 16.2%; even the best spectral model (FourierGNN) rises by about 20%. Please correct these numbers and clarify whether 'Spectral GNN' refers to a category or a specific model.","section":"§4, RQ1"},{"comment":"The manuscript relies on promised but absent artifacts. The dataset link is given as '[Real-E Link]', the benchmark link as '[Benchmark Link]', and preprocessing/splitting details are deferred to 'Link' and 'Github'. Section 2 only says 'we excluded time series with substantial missing values'; Section 4 reports a hyperparameter search range but not the actual per-model configurations, normalization, or validation procedure. For a benchmark paper, this prevents verification of the central empirical claims. Please include a complete data card, preprocessing script, and per-model hyperparameter settings in the submission or an accessible appendix.","section":"§2, §4, 'Real-E Link', 'Benchmark Link', 'Link', 'Github'"}],"minor_comments":[{"comment":"The summation indices in Eq. (1) are inconsistent: the first sum runs from t=0 to T-1 and the second from t=1 to T. Please define T and align the index ranges.","section":"Eq. (1)"},{"comment":"The top row of Table 2 (0.5391, 0.6245, 1.0969, 0.9698) is unlabeled. Please state explicitly whether these are TGV or GSD values and for which correlation-window settings.","section":"Table 2"},{"comment":"Reference [22] and [23] are duplicate Informer entries; [22] has a malformed author field ('H. et al. Zhou'). Please merge and correct.","section":"References [22], [23]"},{"comment":"The abstract says 'over 74 power stations across 30+ European countries,' while Table 1 and Figure 1 report 39 countries and 20 energy types. Please reconcile these numbers.","section":"Abstract and Section 2"},{"comment":"There are spacing/grammar typos, e.g., 'TheReal-E' in the abstract and Section 2, and 'our dataset lay a solid foundation' in Section 5. Please proofread.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a five-page CIKM submission with several load-bearing details deferred to placeholder links. The core idea and dataset are promising, but the empirical foundation needs to be self-contained and internally consistent. I did not find evidence of misconduct; the issues are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Real-E is a useful resource in principle: a 39-country, 20-energy-type, 10-year compilation from ENTSO-E with rich metadata, plus two simple metrics (TGV, GSD) for tracking correlation shifts. That part of the paper is worth keeping. The problem is the empirical story. The benchmark experiments run only on a Germany/France subset, yet the abstract and conclusions generalize to the full 39-country dataset. Nothing in the text shows Germany and France represent the whole. The stress-test note is right that TGV/GSD comparisons across datasets are confounded because Frobenius norms scale with matrix dimension. I also can't reproduce the headline degradations from Table 2. The transformer average goes from about 0.117 to 0.215, an ~84% increase, not 85.4%, and the '16.20%' spectral GNN degradation is wrong by a wide margin—look at FourierGNN and StemGNN, they degrade far more. The TPGNN* table cell is corrupted. And the paper still points to placeholder links for the dataset and code. For a benchmark paper, that's the load-bearing gap: the artifact isn't verifiable from this manuscript.\n\nCredit where it's due: the dataset design is sensible, the seasonal correlation plots are informative, and the idea of testing whether current models break under real cross-energy dynamics is exactly the right question. The section on metrics is straightforward but legitimate.\n\nWho should read this: anyone working on multivariate forecasting for power systems, especially if they want a broader benchmark than ETT/Solar. The dataset, once actually released, could be a genuine contribution.\n\nMy recommendation: this deserves serious peer review, but only if the authors ship the repository and fix the arithmetic. The two-country to 39-country leap must be either defended with evidence or removed from the conclusions. If a final version has done that, it's a fine benchmark paper. As submitted, the preprint overclaims and the verification burden is unmet.","headline":"Useful dataset resource, but the benchmark claims in this preprint rest on a two-country subset and some sloppy arithmetic.","tokens_in":8288,"tokens_out":2363,"would_cite":false,"duration_ms":26790,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Real-E, a 39-country, 20-source electricity dataset, exposes why current forecasters fail to generalize, and the paper argues that the culprit is the non-stationary correlation structure between energy sources.","keywords":["electricity forecasting","multivariate time series","benchmark dataset","non-stationary correlation","graph neural networks","transformers","energy mix","generalization"],"falsifier":"Compute TGV and GSD on the full Real-E collection (all 39 countries) rather than the Germany–France subset; if the subset's volatility is systematically higher or lower than the full dataset, the claim that real-world correlation dynamics are more volatile and therefore break transformers may be an artifact of the subset. Similarly, test a model that ranks well on the subset on a country with a different energy mix, such as hydro-dominated Norway; if its rank reverses, the generalizability claim fails.","tokens_in":7312,"feed_emoji":"⚡","tokens_out":6408,"duration_ms":61709,"temperature":0.7,"pith_summary":"This paper introduces Real-E, a real-world electricity forecasting dataset spanning 39 European countries, 20 generation types, and up to ten years of 15-minute records. The authors argue that existing benchmarks, typically single-country and single-source series, hide how much the correlations between energy sources shift over time. To expose that, they define two metrics, TGV and GSD, and show Real-E has roughly twice the correlation volatility of earlier benchmarks. Benchmarking over 20 models, they find that Transformer-based and spectral-GNN forecasters degrade sharply on Real-E, while spatial-GNN models that explicitly adapt graph structure generalize best. The contribution is a dataset and a diagnosis: models that rely on stationary cross-series structure will not hold up in real grids.","feed_headline":"39-country grid data makes transformer forecasters stumble","feed_subtitle":"Real-E shows cross-energy correlations shift too fast for global-attention models; spatial GNNs adapt and win.","key_machinery":"The central object is Real-E, a multivariate time-series dataset built from the ENTSO-E transparency platform, organized by country and generation type with rich metadata. The paper's argument runs through two new diagnostic metrics: Temporal Graph Volatility (TGV), the Frobenius norm of the difference between adjacency matrices of time-windowed correlation graphs, and Graph Spectral Divergence (GSD), the Euclidean distance between sorted Laplacian spectra of those graphs. These metrics quantify how quickly and how structurally the cross-energy dependencies change, and they do the work of showing that Real-E is more volatile than existing benchmarks. The forecasting comparison itself is the","core_discovery":"The paper's central discovery is that the cross-energy correlation structure of real European electricity systems is non-stationary at the scale of ordinary forecasting windows, and that this non-stationarity, not just noise or missing data, is what breaks state-of-the-art time series models. Using Real-E, a dataset built from the European network operators' transparency data, the authors construct time-windowed Pearson correlation graphs between generation types and quantify their evolution with two metrics: Temporal Graph Volatility, a Frobenius-norm difference between adjacency matrices of consecutive windows, and Graph Spectral Divergence, the Euclidean distance between sorted Laplacian","pith_inferences":["The paper's comparison uses a Germany–France subset of Real-E; if TGV/GSD computed across all 39 countries show even higher volatility, the reported model rankings may understate the generalization gap.","A practical extension would be to measure TGV/GSD at changing window lengths to test whether the volatility is an artifact of 15-minute sampling or reflects genuine regime shifts; the metric threshold that separates benign from breaking correlation dynamics is not yet established.","Because Real-E includes load, balancing, market, and transmission tables, the same benchmark protocol could be applied to forecasting prices or imbalances, not just generation; the paper's conclusion that spatial GNNs win on generation may or may not transfer to those targets.","The dataset's 15-minute resolution and ten-year span also invite long-horizon and hierarchical forecasting benchmarks (country to bidding zone to unit), which the paper only starts to explore."],"forward_implications":["If Real-E is representative of real grids, forecasting models intended for deployment should be validated on multi-country, multi-source data rather than single-series benchmarks.","Transformer and spectral-GNN architectures that assume stable cross-series correlations will not transfer to electricity systems; the paper says global attention mechanisms struggle with rapidly shifting dependencies.","Spatial GNNs that explicitly model and adapt the graph structure offer a stronger starting point for robust electricity forecasting.","TGV and GSD could be used as dataset selection tools: a benchmark with low volatility may give misleadingly optimistic generalization estimates.","The 85.4% average error increase across Transformer baselines quantifies the gap between benchmark performance and realistic grid conditions."],"supporting_citations":[{"why":"Supplies the raw European transmission and generation data Real-E is built from.","marker":"[11]"},{"why":"The UCI Electricity series used as a single-country, single-type benchmark comparison.","marker":"[1]"},{"why":"The Solar dataset and the LSTNet baseline, both part of the comparison.","marker":"[8]"},{"why":"The ETT benchmark datasets, used to show Real-E has far more complex correlation structure.","marker":"[23]"},{"why":"Graph WaveNet, the spatial-GNN baseline that generalizes best on Real-E.","marker":"[18]"},{"why":"MTGNN, another spatial-GNN baseline that keeps its ranking on Real-E.","marker":"[17]"},{"why":"FourierGNN, the spectral-GNN method that leads prior benchmarks but degrades on Real-E.","marker":"[19]"},{"why":"TGGC, a spectral-temporal GNN whose performance collapses on Real-E.","marker":"[6]"},{"why":"Autoformer, a Transformer baseline whose error rises sharply on Real-E.","marker":"[15]"},{"why":"FEDformer, another Transformer baseline whose error rises sharply on Real-E.","marker":"[24]"}],"fun_headline_variants":["Grid correlations shift too fast for transformer forecasters","Real-E benchmark: non-stationary links break forecasters","European grid data challenges state-of-the-art time series models","New dataset reveals why forecasting models stumble on real grids","Time-varying grid correlations expose model limits"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper's conclusions assume that the Germany and France generation-by-type data at 15-minute resolution is representative enough of Real-E's full 39-country, 20-source collection that the measured model rankings and volatility metrics on this subset carry over to the whole dataset.","fun_headline_variants_meta":{"raw":{"variants":["Grid correlations shift too fast for transformer forecasters","Real-E benchmark: non-stationary links break forecasters","European grid data challenges state-of-the-art time series models","New dataset reveals why forecasting models stumble on real grids","Time-varying grid correlations expose model limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1321,"prompt_tokens":646,"completion_tokens":675,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":390,"completion_tokens_details":{"reasoning_tokens":600}},"tokens_in":390,"tokens_out":675,"duration_ms":7974,"temperature":1.0,"reasoning_tokens":600,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:56:05.344758+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute TGV and GSD on the full Real-E collection (all 39 countries) rather than the Germany–France subset; if the subset's volatility is systematically higher or lower than the full dataset, the claim that real-world correlation dynamics are more volatile and therefore break transformers may be an artifact of the subset. Similarly, test a model that ranks well on the subset on a country with a different energy mix, such as hydro-dominated Norway; if its rank reverses, the generalizability claim fails.","supporting_citations":[{"cited_title":"Entso-e electricity data repository, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the raw European transmission and generation data Real-E is built from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Solar dataset and the LSTNet baseline, both part of the comparison."},{"cited_title":"Informer: Beyond efficient transformer for long sequence time-series forecasting.Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):11106–11115, 2020","cited_arxiv_id":null,"evidence_quote":"The ETT benchmark datasets, used to show Real-E has far more complex correlation structure."},{"cited_title":"Graph wavenet for deep spatial-temporal graph modeling","cited_arxiv_id":null,"evidence_quote":"Graph WaveNet, the spatial-GNN baseline that generalizes best on Real-E."},{"cited_title":"Connecting the dots: Multivariate time series forecasting with graph neural networks","cited_arxiv_id":null,"evidence_quote":"MTGNN, another spatial-GNN baseline that keeps its ranking on Real-E."},{"cited_title":"FourierGNN: Rethinking multivariate time series forecasting from a pure graph perspective","cited_arxiv_id":null,"evidence_quote":"FourierGNN, the spectral-GNN method that leads prior benchmarks but degrades on Real-E."},{"cited_title":"Towards Expressive Spectral-Temporal Graph Neural Networks for Time Series Forecasting","cited_arxiv_id":"2305.06587","evidence_quote":"TGGC, a spectral-temporal GNN whose performance collapses on Real-E."},{"cited_title":"Autoformer: Decomposition transformers with autocorrelation for long-term series forecasting.Ad- vances in Neural Information Processing Systems (NeurIPS), 2021","cited_arxiv_id":null,"evidence_quote":"Autoformer, a Transformer baseline whose error rises sharply on Real-E."},{"cited_title":"Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting","cited_arxiv_id":null,"evidence_quote":"FEDformer, another Transformer baseline whose error rises sharply on Real-E."}],"review_version":1}