{"id":"7eb72549-dbe2-4b12-a630-fe06be78a93c","arxiv_id":"2505.01959","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An ensemble of LightGBM, CatBoost, and a neural network forecasts grid carbon intensity for 4 days and beats CarbonCast by about 20% average MAPE across 11 grids.","lead":"EnsembleCI combines three standard machine learning models into an ensemble to forecast electricity carbon intensity up to four days ahead, and reports about 20% lower forecast error than the previous CarbonCast method. A smart generalist should read it because better carbon intensity forecasts can tell datacenters and power users when to shift compute to cleaner hours.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed gain over CarbonCast may stem from an unvalidated baseline modification: Section 4.1 removes the one-day OASIS/ENTSO-E source-production forecasts that CarbonCast's published two-tier pipeline is designed to consume.","rationale":"The paper's central claim is that EnsembleCI surpasses the state-of-the-art CarbonCast by an average of 19.58% in MAPE. The load-bearing condition is that the CarbonCast baseline is the real published system and that the comparison is fair. Section 4.1 modifies that baseline by removing one-day external source-production forecasts from OASIS and ENTSO-E. The publisher's CarbonCast pipeline is explicitly two-tier and consumes source-production information; the paper does not show that this removal is neutral for CarbonCast, and the ensemble models never relied on those external forecasts. This is a systematic confound rather than a random error, and it directly affects the reported day-1 improvement and the headline average. A single reproduction with the original CarbonCast data pipeline would settle whether the concern lands. The reader's CONDITIONAL verdict is appropriate; this pass does not change it. The absence of error bars and modern baselines are secondary concerns but do not displace the baseline-fairness issue as the most load-bearing point.","tokens_in":8622,"tokens_out":9525,"duration_ms":86724,"concrete_test":"Re-run the evaluation with the original CarbonCast v3.0 code and do not delete the OASIS/ENTSO-E one-day source-production inputs, keeping the same train/test split, grids, and MAPE metric. Compare the resulting day-1 MAPE per grid to Table 2's CarbonCast column. If the original CarbonCast average day-1 MAPE falls below 7.05%, recompute the per-grid relative improvements and the aggregate 19.58% figure; if the revised average drops materially or changes sign on grids like DE, the central claim overstates EnsembleCI's advantage. Also report original-CarbonCast versus modified-CarbonCast MAPE to quantify the handicap introduced by the removal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table 2 is not against the published CarbonCast baseline. Section 4.1 states: \"we remove the one-day source production predictions from external sources like OASIS and ENTSO-E for both CarbonCast and EnsembleCI.\" For CarbonCast (refs 26, 27), those day-ahead source-production forecasts are part of the documented two-tier pipeline; its Tier-1 model is designed to forecast source mixes, and near-term operator forecasts are among the most informative inputs for day-1 carbon intensity. EnsembleCI's sublearners were selected on the reduced feature set and never consume these external forecasts, so the removal is not neutral: it strips a feature that only the baseline was designed to exploit. The reported day-1 gain (18.1%) is exactly the horizon where external source forecasts are most accurate and most valuable, and the headline 19.58% average includes that horizon. The paper contains no ablation quantifying CarbonCast's sensitivity to this removal, so the observed advantage may reflect a handicapped baseline rather than ensemble superiority.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EnsembleCI, an AutoGluon-based multi-layer stacking ensemble of LightGBM, CatBoost, and a fastai tabular neural network for forecasting grid carbon intensity up to four days ahead. The method is evaluated on 11 US and EU regional grids using temporal train/test splits, with MAPE as the primary metric. The authors report that EnsembleCI achieves the lowest MAPE in 10–11 grids across forecast days 1–4 and an average relative improvement of 19.58% over CarbonCast, along with lower MAPE degradation over longer horizons. The paper also presents permutation feature importance for three grids to support interpretability claims.","tokens_in":8836,"tokens_out":4254,"duration_ms":44022,"significance":"If the empirical claims hold, EnsembleCI is a practically useful contribution to carbon-aware scheduling, since accurate multi-day carbon intensity forecasts are directly relevant to datacenter load shifting. Strengths of the paper include proper temporal train/test splits that avoid target leakage, averaging over five runs, an open-source release of code and data, and a concrete feature-importance analysis. The central empirical claim, however, depends on the fairness of the CarbonCast comparison, and the current reporting does not provide statistical support for the claimed dominance. The paper is a short conference-style study; the underlying method is reasonable, but the evaluation needs strengthening before the headline accuracy gains can be accepted.","major_comments":[{"comment":"The comparison against CarbonCast is not demonstrably neutral. Section 4.1 states that the authors remove the one-day source production predictions from external sources such as OASIS and ENTSO-E for both CarbonCast and EnsembleCI. According to Figure 1 and Section 2.1, CarbonCast's two-tier architecture is explicitly designed to consume source forecasts: Tier 1 produces source forecasts and Tier 2 consumes them along with weather and historical CI. Removing the external day-ahead source production inputs may therefore handicap only CarbonCast, particularly at the day-1 horizon where such operator forecasts are most informative. The paper provides no ablation quantifying CarbonCast's sensitivity to this removal, and no results for the published CarbonCast pipeline. Because the headline 19.58% average improvement includes the day-1 horizon, the reported advantage over CarbonCast in Table 2 could overstate EnsembleCI's real benefit. Please add an ablation reporting CarbonCast with and without the external source production predictions, or otherwise demonstrate that the removal is neutral for the baseline.","section":"4.1 Methodology and Figure 1"},{"comment":"The evaluation reports only MAPE point estimates. The authors state that five runs are averaged, but no standard deviations, confidence intervals, or significance tests are reported for any grid or horizon. Without measures of variability, the claims that EnsembleCI 'consistently surpasses' CarbonCast and that it is 'more robust' in long-term forecasting are not statistically supported. For example, the day-3 and day-4 differences in some grids are small (e.g., MISO day-3: 7.65 vs. 7.75; NL day-4: 9.33 vs. 13.31 is larger, but variance is unknown). Please report per-grid error bars from the five runs and, where appropriate, a paired significance test across grids or horizons.","section":"Table 2 and Section 4.2"},{"comment":"The recursive design of the day-2-to-4 model is described only briefly. The text says the day-1 model outputs, together with future weather forecasts and datetime features, are inputs to the day-2-to-4 model, which is applied recursively for three iterations. It is not specified whether the day-2-to-4 model is trained on teacher-forced ground-truth day-1 values or on the day-1 model's own predictions, and it is unclear whether the recursive application introduces error accumulation that is specific to this design. Please clarify the training procedure and, ideally, report an ablation comparing direct multi-day forecasting with the recursive scheme.","section":"Section 3, Implementation"}],"minor_comments":[{"comment":"The caption contains the word 'blue' in 'up to 4 days blue into the future'; this appears to be a leftover editing artifact and should be removed.","section":"Table 2 caption"},{"comment":"The text contains a typo: 'regional girds' should be 'regional grids'.","section":"Section 1"},{"comment":"The subsection heading 'Subleaner selection' and the body text contain repeated misspellings of 'sublearner'.","section":"Section 3"},{"comment":"The x-axis label 'ERCO' should be 'ERCOT' to match the grid name used elsewhere.","section":"Figure 4"},{"comment":"The source of the weather forecast features is not specified. For reproducibility, please state which dataset provides the 24-hour weather forecasts and how they are aligned with the grid timestamps.","section":"Section 4.1"},{"comment":"The 'CarbonCast v3.0 dataset' is not defined or cited. Please specify its contents, version, and how it was obtained.","section":"Section 4.1"},{"comment":"The heading 'Top-3 features for three grids with the highest average ranks for three grids' is awkward and should be rephrased, for example as 'Top-3 features for three exemplar grids'.","section":"Table 3"},{"comment":"Figures 2 and 4 show point estimates only; adding error bars or shaded confidence bands would make the claimed robustness and variability reductions visible.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is incremental but suitable for a venue like E-ENERGY if the evaluation is made credible. The main risk is that the headline comparison to CarbonCast may be based on a modified baseline whose sensitivity to the modification was never tested. I would ask the authors to run the CarbonCast ablation with and without external source production predictions before resubmission, and to add variance reporting. If the ablation shows that the removal is neutral, the paper could be acceptable after minor revision; without it, the central empirical claim remains unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the core result is plausible but the headline number is shakier than it looks. EnsembleCI, a stacking ensemble of LightGBM, CatBoost, and a small NN, beats CarbonCast on MAPE over 1-4 day horizons across 11 US/EU grids, with 18-23% relative improvement. The evaluation design is mostly sound: temporal splits, five-run averages, per-grid and per-horizon reporting. The recursive day-2-4 forecasting and the permutation-importance analysis are nice touches.\n\nThe biggest soft spot is exactly the one flagged in the stress test: Section 4.1 removes the one-day OASIS/ENTSO-E source-production forecasts from both models. CarbonCast's published two-tier pipeline is built around near-term source forecasts; stripping that input likely hurts CarbonCast more than EnsembleCI, whose sublearners were never given those features. The paper offers no ablation showing CarbonCast's sensitivity to the removal, so we can't tell how much of the 19.58% average improvement is due to a handicapped baseline. That said, the same section shows LightGBM, CatBoost, and NN also beat CarbonCast under the same reduced feature set, which suggests CarbonCast is weak even without the external forecasts. But the day-1 gap is exactly where the removed forecasts matter most, so the concern is not trivial.\n\nOther soft spots are minor: only MAPE is reported, no error bars or significance tests; no other modern baselines (e.g., transformer time-series models); feature importance is only shown for three grids. The paper doesn't overclaim; it explicitly says performance varies across grids. The GitHub link promises code and data, which would help verification.\n\nBottom line: a useful applied study, but the central comparison needs a robustness check. If I were refereeing, I'd ask for CarbonCast with and without external source forecasts, error bars, and at least one more baseline before accepting. The paper is worth engaging with and deserves peer review; it's just not there yet.","headline":"A plausible applied win for stacked ensembles in carbon-intensity forecasting, but the headline comparison to CarbonCast is muted by an unvalidated baseline modification.","tokens_in":9346,"tokens_out":2264,"would_cite":false,"duration_ms":23338,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An ensemble of three learners predicts grid carbon intensity up to four days ahead and beats the previous best method by 19.58% on average.","keywords":["carbon intensity forecasting","ensemble learning","multi-day forecasting","stacking","grid carbon intensity","mean absolute percentage error","feature importance","load shifting"],"falsifier":"Restore CarbonCast's tier-1 source-production forecasts and rerun the same 11-grid, four-day evaluation; if the day-1 average MAPE gap shrinks below the reported 18.1%, or the four-day gap closes, the ensemble advantage depends on the input modification rather than on ensembling. A cleaner refutation would be any grid where the weighted ensemble's day-4 MAPE exceeds every one of its own sublearners' day-4 MAPE by more than noise, since the stacking stage would then be adding error rather than removing it.","tokens_in":8443,"feed_emoji":"⚡","tokens_out":8146,"duration_ms":77738,"temperature":0.7,"pith_summary":"Carbon intensity — the grams of CO₂-equivalent emitted per kilowatt-hour of grid electricity — is the quantity a carbon-aware scheduler needs to see in advance to shift compute to cleaner hours. The paper argues that a single fixed architecture trained once for all grids cannot supply that signal reliably, because grids differ sharply in energy mix and weather sensitivity. It proposes EnsembleCI, which trains a pool of three strong predictors per grid and merges them through weighted stacking, and reports that this ensemble beats the prior state-of-the-art method in 10 of 11 grids on day 1 and 11 of 11 on days 2–4, with an average 19.58% lower mean absolute percentage error. It also claims the ensemble degrades more gracefully over a four-day horizon, with average error growth of 3.63 points versus 5.43 for the baseline, and that its feature importances reveal sensible region-specific drivers. The practical upshot is a cheaper, more trustworthy four-day carbon forecast that data centers could act on.","feed_headline":"One ensemble beats CarbonCast on 4-day carbon forecasts","feed_subtitle":"Across 11 grids, average MAPE drops from 12.48% to 9.56% by day four — a 19.58% gain.","key_machinery":"The load-bearing mechanism is multi-layer stacking over a deliberately small and diverse pool of sublearners. A base stage trains three strong tabular learners — two gradient-boosted tree methods and a fully connected neural network — on raw features; a stacking stage concatenates the raw features with the base models' predictions and retrains the same learners on that enlarged input; and ensemble selection assigns the final weights to the stacked predictions. Two separate trained models create the 96-hour output: one predicts the first 24 hours from historical source production, weather, and datetime features, and a second receives the day-1 outputs plus the same auxiliary features and is applied recursively for days 2–4. This two-stage composition is what carries the paper's argument: the per-grid weights absorb regional energy-mix differences that a fixed two-tier architecture cannot express, while the recursion slows error accumulation at long horizons.","core_discovery":"EnsembleCI's claim is that regional adaptability, not a larger or deeper single model, is what unlocks accurate multi-day carbon-intensity forecasting. Rather than fit one architecture to all regions, it trains three complementary predictors on each grid's raw features — historical source-generation mix, historical carbon intensity, weather forecasts, and datetime encodings — then builds a second stacking layer on the concatenation of raw features and base predictions, and finally aggregates the stacked outputs with ensemble-selection weights. On the CarbonCast dataset covering six US and five EU grids, the ensemble averages 5.92%, 8.16%, 9.08%, and 9.56% MAPE for days 1–4, against 7.05%, 9.62%, 11.12%, and 12.48% for CarbonCast, which the paper reports as relative improvements of 18.1%, 17.13%, 19.69%, and 23.4%. The paper additionally reports steadier long-horizon behavior — a day-1-to-day-4 average MAPE increase of 3.63 points versus 5.43 — and permutation-importance analyses in which the three sublearners agree on each grid's dominant drivers, such as solar and irradiance in solar-heavy CISO and DE, and coal and wind in MISO. On the paper's own terms, the ensemble is both the more accurate and the more interpretable tool.","pith_inferences":["Editorial inference: the reported 19.58% gain may be sensitive to removing the one-day source-production forecasts from both models; a rerun with CarbonCast's native tier-1 forecasts restored is the cheapest way to see whether the ensemble's edge survives a fully native input set.","Editorial inference: the period-retraining suggestion is untested, so a rolling-window experiment tracking weights and MAPE across seasons would reveal how quickly per-grid ensembles go stale as renewable penetration grows.","Editorial inference: the same base-stack-weight pipeline could be lifted to other grid variables such as locational prices or renewable generation, where heterogeneous regional regimes are equally central."],"forward_implications":["Day-ahead forecasts near 5.9% average MAPE are accurate enough to feed into workload-shifting and battery-scheduling policies that today's CarbonCast-level errors can defeat.","The advantage grows with horizon — from 18.1% on day 1 to 23.4% on day 4 — so multi-day planning, not just hour-ahead shifting, is where the ensemble pays off.","Grid-specific feature rankings give operators a diagnostic: when the model leans on solar and irradiance, it flags renewable-driven variability; when it leans on coal and wind, it flags the fuel mix that dominates emissions.","Because the architecture trains per grid with a shared procedure, adding a new grid or retraining after a fuel-mix change does not require redesigning the network."],"supporting_citations":[{"why":"Supplies the CarbonCast baseline, its two-tier architecture, and the dataset split used for the comparison.","marker":"[26, 27]"},{"why":"One of the three sublearners, a gradient-boosted tree method shown to beat CarbonCast before ensembling.","marker":"[24]"},{"why":"The second sublearner, a boosting model that handles categorical features well.","marker":"[14]"},{"why":"The third sublearner, a fully connected network built on a general tabular deep-learning library.","marker":"[22]"},{"why":"The automated machine-learning framework the implementation builds on for stacking and model selection.","marker":"[17]"},{"why":"The multi-layer stacking strategy that EnsembleCI adopts for its two-stage architecture.","marker":"[30]"},{"why":"The ensemble-selection algorithm used to produce the final weighted aggregation of stacked predictions.","marker":"[10]"},{"why":"Permutation importance, the method used in Section 4.3 to identify grid-specific key features.","marker":"[5]"},{"why":"The Electricity Maps data that define the 11 grids and their energy-source mixes in Table 1 and Figure 3.","marker":"[1]"}],"fun_headline_variants":["EnsembleCI improves carbon forecast accuracy by 19.58%","Regional ensemble beats CarbonCast on 11 grids","Stacked ensemble cuts MAPE, boosts long-term robustness","EnsembleCI: lower error, steadier multi-day carbon forecasts","Adaptive ensemble wins over CarbonCast for carbon intensity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the comparison is fair even though the paper removes the external one-day source-production forecasts that CarbonCast's two-tier design was built to consume, so any handicap from that removal is assumed to hit both models equally.","fun_headline_variants_meta":{"raw":{"variants":["EnsembleCI improves carbon forecast accuracy by 19.58%","Regional ensemble beats CarbonCast on 11 grids","Stacked ensemble cuts MAPE, boosts long-term robustness","EnsembleCI: lower error, steadier multi-day carbon forecasts","Adaptive ensemble wins over CarbonCast for carbon intensity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1662,"prompt_tokens":1051,"completion_tokens":611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":529}},"tokens_in":667,"tokens_out":611,"duration_ms":5533,"temperature":1.0,"reasoning_tokens":529,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:05:25.944902+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Restore CarbonCast's tier-1 source-production forecasts and rerun the same 11-grid, four-day evaluation; if the day-1 average MAPE gap shrinks below the reported 18.1%, or the four-day gap closes, the ensemble advantage depends on the input modification rather than on ensembling. A cleaner refutation would be any grid where the weighted ensemble's day-4 MAPE exceeds every one of its own sublearners' day-4 MAPE by more than noise, since the stacking stage would then be adding error rather than removing it.","supporting_citations":[{"cited_title":"LightGBM: A highly efficient gradient boosting decision tree","cited_arxiv_id":null,"evidence_quote":"One of the three sublearners, a gradient-boosted tree method shown to beat CarbonCast before ensembling."},{"cited_title":"Fastai: a layered api for deep learning","cited_arxiv_id":null,"evidence_quote":"The third sublearner, a fully connected network built on a general tabular deep-learning library."},{"cited_title":"Multi-layer stacking ensemble learners for low footprint network intrusion detection","cited_arxiv_id":null,"evidence_quote":"The multi-layer stacking strategy that EnsembleCI adopts for its two-stage architecture."},{"cited_title":"Ensemble selection from libraries of models","cited_arxiv_id":null,"evidence_quote":"The ensemble-selection algorithm used to produce the final weighted aggregation of stacked predictions."},{"cited_title":"Permutation importance: a corrected feature importance measure","cited_arxiv_id":null,"evidence_quote":"Permutation importance, the method used in Section 4.3 to identify grid-specific key features."},{"cited_title":"https://app.electricitymaps.com/map/","cited_arxiv_id":null,"evidence_quote":"The Electricity Maps data that define the 11 grids and their energy-source mixes in Table 1 and Figure 3."}],"review_version":1}