REVIEW 4 major objections 5 minor 33 references
Causality analysis of electricity market liberalization on electricity price using novel Machine Learning methods
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Opening US state electricity markets to competition caused residential prices to fall by about 0.795 cents per kWh (≈7%) in the two years after individual producers entered, according to a causal machine-learning analysis.
desk verdict A useful methodological comparison of causal ML forecasting methods on short panels, but the headline 7% price drop is built on a timing assumption that likely mixes pre- and post-treatment months. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The analysis is carried by a counterfactual forecasting mechanism in the synthetic-control family. A model is trained globally on the pre-intervention data of both treated and control states, with lagged electricity prices and external covariates (state-level income for demand, natural gas price for supply) as inputs, and then forecasts what each treated state's price would have been after 1998 absent liberalization. The ATT is the difference between the observed price and this predicted counterfactual, averaged over treated units and post-intervention time. The validity of this mechanism rests on two assumptions stated in the paper: the pre-intervention relationship between prices and covariates continues after the intervention (Assumption 2, Equations 3–4), and time-varying confounders do not affect treated and control units differently; the paper itself notes that the presence of such confounders invalidates the model. DeepProbCP—the paper's preferred implementation—uses an LSTM layer with a moving-window, seasonal-exogenous strategy and is trained with quantile loss and CRPS, giving it an edge on the short, many-series panels examined here.
What would settle it
Re-run the analysis with a placebo treatment assigned to non-liberalized states at the same calendar date (1998–1999), using the same model and covariates. If the placebo ATT for these untreated states is as large, in absolute value, as the −0.795 ¢/kWh estimated for the liberalized states, then the effect is not specific to liberalization and the causal claim fails.
Extended reading notes
Core claim
On the paper's own terms, its central discovery is a causal claim: US electricity market liberalization—timed by the entry of individual producers rather than by the year of legislative approval—caused a short-term reduction in residential electricity prices. Across the eight states where individual producers' market share spiked in 1998–1999, the average treatment effect on the treated estimated by the DeepProbCP model is −0.795 ¢/kWh over the two post-intervention years, a 7% decrease relative to the average price in the year before the intervention. The paper further claims that its preferred model, a global non-parametric counterfactual forecaster, is the most reliable of the four examined for this kind of intervention, and that all models pass the placebo test, so the sign and ordering of the effect estimate are consistent across specifications.
Load-bearing premise
The entire estimate depends on the assumption that, had liberalization never occurred, each treated state's electricity price would have continued to follow the same relationship to its past prices, income, and gas prices observed before 1998, and that nothing else changed these states differently from the controls at the same moment.
Editorial extensions
If this is right
- Residential prices in liberalized US states fell by about 7% within two years of individual producers' entry, implying a short-term consumer benefit from retail competition.
- Dating liberalization by producer entry rather than by legislative approval changes the estimated effect and should become standard in evaluating such interventions.
- The DeepProbCP global forecasting framework appears better suited to short, many-series policy panels than TSMixer, ASCM, or Causal ARIMA.
- All four models pass the placebo test and estimate a negative treatment effect, so the direction of the price reduction is robust across specifications, though the magnitude varies from roughly −0.44 to −1.06 ¢/kWh.
- The paper's conclusions are limited to a two-year window and say nothing about whether the price reduction persists after 1999.
Reading between the lines
- If liberalization reduces prices only in the short term, as both this paper and earlier difference-in-differences work suggest, the consumer gains may be a transition phenomenon rather than a durable benefit; a longer-horizon replication with the same causal ML framework could settle this.
- The range of ATT estimates across models (from −0.44 to −1.06 ¢/kWh) implies that policy conclusions from a single causal ML model should be treated with caution, and ensembles of estimators might give a more stable basis for policy decisions.
- The intervention-timing correction (using the year of producer entry rather than policy approval) could improve causal ML evaluations of deregulation in other sectors, such as natural gas or telecommunications, where implementation lags legislation.
- The probabilistic forecasting capability that the authors note in DeepProbCP could be used to construct prediction intervals around the ATT, giving policymakers a measure of uncertainty that the current analysis does not report.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper applies four counterfactual forecasting methods (DeepProbCP, TSMixer, ASCM, and Causal ARIMA) to estimate the short-term causal effect of electricity market liberalization on residential electricity prices in US states. Using state-level price data from 1990 to 1999, with income and gas prices as covariates, the authors define treatment by the first spike in individual producers' market share and set 1998-1999 as the post-intervention window. They report that DeepProbCP performs best on synthetic ground-truth experiments and on control-unit holdout error, and they estimate an ATT of -0.795 cents/kWh, described as a 7% price reduction relative to the pre-intervention year. The paper also argues for causal machine learning as a useful approach for energy-policy intervention analysis.
Significance. If the estimate is valid, the paper provides policy-relevant evidence that retail competition produced a short-term residential price reduction in treated US states. The study has notable strengths: the counterfactual predictions are genuinely out-of-sample forecasts from pre-intervention data, the synthetic experiments have known treatment effects and support the choice of DeepProbCP, placebo tests on control units are reported, and the code is publicly available. The main limitations are the timing definition of treatment, the absence of uncertainty quantification for the headline ATT, and the reliance on control-unit fit for model selection; these affect the precision and interpretation of the central claim but do not undermine the value of the comparison exercise.
major comments (4)
- [§4.1, Figure 4, Table 4] The treated group is defined as states whose individual-producer share "jumped in 1998-1999," yet the post-intervention period is set uniformly to 1998-1999. For a state whose first spike occurred in 1999, the entire 1998 calendar year is a pre-treatment year, so the pooled ATT over 1998-1999 mixes post-treatment and no-treatment periods. This biases the reported -0.795 cents/kWh toward zero if the effect is a price decline, and it makes the headline claim of "2 years following liberalization" inaccurate for late-entry states. The paper never reports the entry year per state or a year-by-year ATT; please provide these, or an event-time alignment, to support the causal interpretation.
- [§5, Table 4] The headline ATT of -0.795 cents/kWh is a single point estimate with no confidence interval or other uncertainty measure. The Wilcoxon p-values in Table 3 are placebo-test results, not uncertainty intervals for the treated-unit counterfactual. Given that the four models' ATT estimates range from -0.441 to -1.064 cents/kWh, the model choice is consequential; please report uncertainty intervals (for example, from DeepProbCP's probabilistic forecasts or a bootstrap) and a sensitivity table showing how the 7% claim changes across models and post-window definitions.
- [§4.1, Figure 3] The text states "Out of the 17 states, in 8 states, the share of individual producers jumped in 1998-1999" and then lists nine states: California, Connecticut, Illinois, Maine, Maryland, New Jersey, New York, Pennsylvania, and Rhode Island. Figure 3's caption also says "Only 8 of them." This numerical inconsistency directly affects the treatment-group definition and the resulting ATT; it must be corrected and the intended treatment group stated unambiguously.
- [§3, Eqs. (3)-(4)] The paper explicitly acknowledges that "time-varying confounders, or a failure to closely match control units in the pre-treatment period, results in the invalidity of the model," but the only covariates included are income, gas price, and past prices. No test is provided for state-specific time-varying shocks, such as transition-cost recovery or restructuring-specific regulatory changes, beyond the control-unit forecast errors. Please address this threat more directly, for example with a pre-treatment placebo distribution or a sensitivity analysis adding additional covariates.
minor comments (5)
- [§4.1, Eq. (7)] The notation \(\sin(2\pi/1 t)\) is ambiguous; if a period-1 daily cycle is intended, write \(\sin(2\pi t)\) or define the period explicitly.
- [§4.1] The phrase "regarded as able to stimulate energy load data" appears to be a typo for "simulate energy load data."
- [§3] The abbreviation "SUTV A" should be "SUTVA" (Stable Unit Treatment Value Assumption).
- [§5, Table 3] The text describes the Wilcoxon test as testing for a "difference in means," but the Wilcoxon rank-sum test compares distributions or locations, not means.
- [§4, Eq. (6)] The seasonal period \(S\) used in the MASE denominator is not defined in the text.
Circularity Check
No significant circularity: counterfactual forecasts are out-of-sample and model selection avoids treated outcomes.
full rationale
The core ATT estimate is not circular. DeepProbCP (and the other models) are trained only on pre-intervention data: the paper states that 'the pre-intervention data of all units is used as training data, and the post-intervention period is used as the test data to be forecasted by the model.' Treated post-intervention outcomes are never used in training or in model selection. Model selection is justified by synthetic experiments with known ground truth and by control-unit forecast errors under the null intervention assumption, not by fitting a parameter to the real treated outcome. The ATT is then computed as observed minus forecast, which is a genuine out-of-sample prediction error rather than a fitted quantity renamed as a prediction. There are no load-bearing self-citations: DeepProbCP is taken from Grecov et al. (a different group) and used via its published code, and no uniqueness theorem or author-overlapping citation is invoked to force the modeling choice. The paper does contain a potential timing concern: it sets 1998-1999 as the common post-period for states whose producer-share spike may have occurred in either 1998 or 1999, and the sentence 'Upon inspecting the price data, we can observe a visible change as the share of individual producers starts to go up' could suggest outcome-informed timing inspection. However, these are identification and design issues, not circular reductions: no equation, fitted parameter, or self-citation makes the estimated effect equal to an input by construction.
Assumptions & free parameters
free parameters (2)
- synthetic trend_rate =
1.00005
- synthetic treatment effect quantile constants =
0.3 sigma, 0.6 sigma, 0.9 sigma, 1.2 sigma, 1.5 sigma
assumptions (5)
- domain assumption Unconfoundedness: conditional on covariates (income, gas price) and pre-treatment outcomes, treatment assignment is independent of potential outcomes.
- domain assumption Assumption 2: the relationship between treated unit outcomes and predictors estimated before the intervention remains the same after the intervention (Eqs. 3 and 4).
- ad hoc to paper No anticipation: policy approval did not change prices until individual producers actually entered, based on inspection of price data.
- domain assumption Consistency and SUTVA: control states are unaffected by liberalization and treated states' observed outcomes equal their treatment potential outcomes.
- ad hoc to paper Synthetic data generation (sum of sine waves plus Gaussian process) resembles real electricity price dynamics enough for model selection.
Cite this review
Pith. "Pith review of Causality analysis of electricity market liberalization on electricity price using novel Machine Learning methods." pith.science (2026). https://pith.science/paper/F6BYP5XV
@misc{pith2026250712331,
author = {Pith},
title = {Pith review of: Causality analysis of electricity market liberalization on electricity price using novel Machine Learning methods},
year = {2026},
howpublished = {\url{https://pith.science/paper/F6BYP5XV}},
note = {Machine review of arXiv:2507.12331}
}
read the original abstract
Relationships between the energy and the finance markets are increasingly important. Understanding these relationships is vital for policymakers and other stakeholders as the world faces challenges such as satisfying humanity's increasing need for energy and the effects of climate change. In this paper, we investigate the causal effect of electricity market liberalization on the electricity price in the US. By performing this analysis, we aim to provide new insights into the ongoing debate about the benefits of electricity market liberalization. We introduce Causal Machine Learning as a new approach for interventions in the energy-finance field. The development of machine learning in recent years opened the door for a new branch of machine learning models for causality impact, with the ability to extract complex patterns and relationships from the data. We discuss the advantages of causal ML methods and compare the performance of ML-based models to shed light on the applicability of causal ML frameworks to energy policy intervention cases. We find that the DeepProbCP framework outperforms the other frameworks examined. In addition, we find that liberalization of, and individual players' entry to, the electricity market resulted in a 7% decrease in price in the short term.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Abadie, A. (2021). Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2):391–425
work page 2021
-
[2]
Administration, U. E. I. (2024). Eia-861m monthly electric power industry report
work page 2024
-
[3]
Allcott, H. (2011). Rethinking real-time electricity pricing. Resource and Energy Economics, 33(4):820–842. Special section: Sustainable Resource Use and Economic Dynamics
work page 2011
-
[4]
Amenta, C., Aronica, M., and Stagnaro, C. (2022). Is more competition better? retail electricity prices and switching rates in the european union. Utilities Policy, 78:101405
work page 2022
-
[5]
Annweiler, C., Fantino, B., and Parot-Schinkel, E. (2011). Alzheimer’s disease - input of vitamin d with memantine assay (ad-idea trial): study protocol for a randomized controlled trial. Trials 12, 230
work page 2011
-
[6]
Athey, S. and Imbens, G. W. (2006). Identification and inference in nonlinear difference-in-differences models. Econometrica, 74(2):431–497
work page 2006
-
[7]
Batalla-Bejerano, J., Costa-Campi, M. T., and Trujillo-Baute, E. (2016). Collateral effects of liberalisation: Metering, losses, load profiles and cost settlement in spain’s electricity system. Energy Policy, 94:421–431
work page 2016
-
[8]
Blagrave, P. and Furceri, D. (2021). The macroeconomic effects of electricity-sector privatization. Energy Economics, 100:105245
work page 2021
Show all 33 references
-
[9]
O., Yoder, N
Chen, S.-A., Li, C.-L., Arik, S. O., Yoder, N. C., and Pfister, T. (2023). TSMixer: An all-MLP architecture for time series forecast-ing. Transactions on Machine Learning Research. Originally available as arXiv:2303.06053
2023 arXiv
-
[10]
Cheng, X., Liu, P., and Zhu, L. (2024). The impact of electricity market reform on renewable energy production. Energy Policy, 194:114306
2024
-
[11]
Eli Ben-Michael, A. F. and Rothstein, J. (2021). The augmented synthetic control method. Journal of the American Statistical Association, 116(536):1789–1803
2021
-
[12]
Erdogdu, E. (2013). Implications of liberalization policies on government support to r and d: Lessons from electricity markets. Renewable and Sustainable Energy Reviews, 17:110–118. 24
2013
-
[13]
Grecov, P., Bandara, K., Bergmeir, C., Ackermann, K., Campbell, S., Scott, D., and Lubman, D. (2021). Causal inference using global forecasting models for counterfactual prediction. In Advances in Knowledge Discovery and Data Mining (PAKDD 2021), LNCS, vol.12713, pages 282–294...
2021
-
[14]
Grecov, P., Prasanna, A., Ackermann, K., Campbell, S., Scott, D., Lubman, D., and Bergmeir, C. (2024). Probabilistic causal effect estimation with global neural network forecasting models. IEEE Transactions on Neural Networks and Learning Systems, 35(4):4999–5013
2024
-
[15]
Glachant, P.L
J.-M. Glachant, P.L. Joskow, M. P. (2022). The evolution of competitive retail electricity markets. Handbook on Electricity Markets, Edward Elgas Publishing pp. 111-155
2022
-
[16]
Joskow, P. L. (2000). Deregulating and regulatory reform in the u.s. electric power sector. MIT Center for Energy and Environmental Policy Research
2000
-
[17]
Joskow, P. L. (2008). Lessons learned from electricity market liberalization. The Energy Journal, 29(2_suppl):9–42
2008
-
[18]
and Branson, Z
Keller, B. and Branson, Z. (2024). Defining, identifying, and estimating causal effects with the potential outcomes framework: A review for education research. Asia Pacific Education Review, 25(3):575–594
2024
-
[19]
J., Nikolova, S., and Sutton, M
Kreif, N., Grieve, R., Hangartner, D., Turner, A. J., Nikolova, S., and Sutton, M. (2016). Examination of the synthetic control method for evaluating health policies with multiple treated units. Health Economics, 25(12):1514–1528
2016
-
[20]
X., and Dustdar, S
Liu, S., Yu, H., Liao, C., Li, J., Lin, W., Liu, A. X., and Dustdar, S. (2022). Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations
2022
-
[21]
Menchetti, F., Cipollini, F., and Mealli, F. (2021). Estimating the causal effect of an intervention in a time series setting: the c-arima approach. arXiv preprint arXiv:2103.06740v3
2021 arXiv
-
[22]
Nagayama, H. (2009). Electric power sector reform liberalization models and electric power prices in developing countries: An empirical analysis using international panel data. Energy Economics, 31(3):463–472
2009
-
[23]
D., López, A., and Salazar-Elena, J
Necoechea-Porras, P. D., López, A., and Salazar-Elena, J. C. (2021). Deregulation in the energy sector and its economic effects on the power sector: A literature review. Sustainability, 13(6):Article 3429. 25
2021
-
[24]
Upton, J
Rose, K., Tarufelli, B., and Gregory B. Upton, J. (2024). Retail electricity market restructuring and retail rates. The Energy Journal, 45(1):1–49
2024
-
[25]
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology,66, 688–701
1974
-
[26]
Rubin, D. B. (1977). Assignment to treatment group on the basis of a covariate. Journal of Educational and Behavioral Statistics, 2(1):1–26
1977
-
[27]
Rubin, D. B. (1986). Comment: Which ifs have causal answers. Journal of the American Statistical Association, 81(396):961–962
1986
-
[28]
Schuelke-Leech, B.-A., Barry, B., Muratori, M., and Yurkovich, B. (2015). Big data issues and opportunities for electric utilities. Renewable and Sustainable Energy Reviews, 52:937–947
2015
-
[29]
Splawa-Neyman, J. (1990). On the application of probability theory to agricultural experiments. Essay on principles. Statistical Science, 5(4):465–480. Translated and edited by D. M. Dabrowska and T. P. Speed
1990
-
[30]
Su, X. (2015). Have customers benefited from electricity retail competition? Journal of Regulatory Economics, 47(2):146–182
2015
-
[31]
Trevino, L. (2008). Liberalization of the electricity market in europe: An overview of the electricity technology and the market place. Working paper, École Polytechnique Fédérale de Lausanne (EPFL). Accessed: 2025-07-12
2008
-
[32]
Zhang, D. (2018). Energy finance: Background, concept, and recent developments. Emerging Markets Finance and Trade, 54(8):1687–1692
2018
-
[33]
and Ji, Q
Zhang, D. and Ji, Q. (2019). Energy finance: Frontiers and future development. Energy Economics, 83:290–292. 26
2019
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.