Pith. sign in

REVIEW 4 major objections 5 minor 58 references

A General Approach of Automated Environment Design for Learning the Optimal Power Flow

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automated environment design via multi-objective hyperparameter optimization consistently outperforms a manually derived baseline on five optimal power flow benchmarks.

desk verdict HPO framing for automated RL environment design is genuinely useful and the OPF evidence is mostly credible; the 'consistently outperforms on five benchmarks' claim overreaches and needs scoping, but this deserves peer review. read the letter →

arxiv 2505.07832 v1 pith:2Z5FRBVQ submitted 2025-05-01 cs.LG cs.AIcs.SYeess.SY

Thomas Wolgast , Astrid Nie{ss}e This is my paper · ORCID
classification cs.LGcs.AIcs.SYeess.SY
keywords reinforcementlearningoptimalpowerflowenvironmentdesignhyperparameteroptimizationmulti-objectiveconstraintsatisfactionrewardOPF-Gym
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the design of a reinforcement-learning environment—reward function, observations, episode length, action scaling, and training data—can be treated as a hyperparameter optimization problem and searched automatically, and that on five optimal power flow (OPF) benchmarks this automated search produces environments that consistently beat a manually derived baseline. This matters because OPF, the nonlinear optimization problem at the core of power-grid operation, is increasingly approximated with RL when solutions are needed too fast for conventional solvers, and environment design choices are known to strongly sway training performance without any consensus on what to choose. The authors build an outer multi-objective optimization loop around an inner RL training loop, scoring each candidate environment on constraint satisfaction and cost error, and reuse established hyperparameter optimization methods. They also run statistical tests on the sampled designs to extract concrete design rules, and verify their gains under longer training, held-out test data, and a second RL algorithm while reporting that the optimized environment can become over-adjusted to the algorithm used during the search.

What carries the argument

The central object is an inner/outer optimization loop: an outer multi-objective optimizer proposes environment designs, each candidate is scored by training a DDPG agent for 40k steps and evaluating it on two metrics—invalid share (constraint satisfaction relative to a conventional OPF solver) and mean valid error (cost suboptimality)—and the search continues toward non-dominated designs. The enabling device is a parameterized environment search space that turns 15 discrete and continuous design choices into hyperparameters, including the continuous reward parameterization ($\beta$ penalty weight, $\psi$ invalid-objective share, valid reward, invalid penalty, Diff-Objective) and the training-data mixture $x,y,z$. This machinery reuses hyperparameter optimization best practices, including a strict train/validation/test split, and lets the authors read design rules off the HPO trace with statistical tests.

What would settle it

Take the exact same environment-design search but change only the inner-loop evaluator to a different RL algorithm or to the full 500k-step budget; if the selected designs no longer dominate the manual baseline under that deployment protocol, the central claim is false. A concrete partial case already appears in the paper: in Economic Dispatch, the design tuned with the short-run DDPG protocol, when retrained with SAC, delivered near-perfect constraint satisfaction but the worst optimization performance of the four combinations—so the decisive test is whether that pattern generalizes across benchmarks.

Watch

Extended reading notes

Core claim

The central discovery is that an RL environment for OPF is not a fixed modeling choice but a point in a design space that can be optimized. The paper defines 15 environment design variables—covering reward composition (penalty weight, invalid-objective share, offsets, diff-objective), training data mixture (realistic, normal, uniform), added observations, episode steps, and action autoscaling—and runs a multi-objective HPO loop (an evolutionary many-objective sampler) with, as inner loop, short training with the deterministic policy-gradient actor-critic algorithm DDPG and evaluation on two metrics: invalid share (constraint violations relative to a conventional solver) and mean valid error (cost suboptimality). On all five benchmark problems, the non-dominated designs from this search either dominate the manually derived baseline design or match it; the paper states that the optimized designs "reproducibly outperform and dominate the baseline design, if we use the same RL algorithm." Statistical screening finds that only a few design decisions consistently matter—notably the Diff-Objective reward, action autoscaling, mixing random data with time-series data, and adding line-load or voltage-magnitude observations—and verification at 500k steps with both DDPG and the soft actor-critic algorithm SAC confirms the gains under the same algorithm, while revealing that the environment can over-adjust to the RL algorithm used in the search.

Load-bearing premise

The method rests on assuming that an environment that wins under the quick screening used during search—short training runs with one reinforcement-learning algorithm—will also win under the real deployment setup with longer training and possibly a different algorithm; the paper's own verification shows one case where this fails.

Editorial extensions

If this is right

  • Practitioners can stop hand-picking OPF environment designs and instead run a multi-objective HPO search, then select a design from the Pareto front according to whether constraint satisfaction or cost optimality matters more.
  • The statistical analysis yields transferable design rules: subtract the uncontrollable cost baseline from the reward (Diff-Objective), use state-dependent action autoscaling, and mix random samples into scarce realistic time-series data; adding line-load or voltage-magnitude observations aids constraint satisfaction.
  • Verified gains persist at 500k training steps and on held-out test data, so the short 40k-step screening used in the outer loop is not merely selecting for fast early learning.
  • The HPO framing means environment search can be merged with agent hyperparameter tuning in one joint optimization, and the paper argues the methodology generalizes to other RL domains.
  • The verification also shows a caveat: an environment tuned for one RL algorithm may underperform under another, so algorithm changes should trigger a re-check of the design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct robustness upgrade the paper leaves untested is to run a small SAC or other-algorithm checkpoint for each candidate during the outer loop and keep only designs that are Pareto-optimal across algorithms; this would specifically counteract the overadjustment it reports.
  • Because only a few of the 15 design variables showed significant effects, the cost of the method could be reduced by a two-stage protocol: use short, high-variance runs to screen out irrelevant variables, then run the full multi-objective HPO on the survivors.
  • The failure pattern in Economic Dispatch suggests a diagnostic worth formalizing: compute the rank correlation between candidate designs' proxy scores and their deployment scores; a low correlation would signal that the proxy is rewarding algorithm-specific artifacts rather than general environment quality.
  • The same split into top/bottom groups with Welch and chi-squared tests could be applied post hoc to any HPO trace, turning an expensive black-box search into a source of interpretable design knowledge.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an automated, multi-objective hyperparameter-optimization (HPO) framework for designing reinforcement-learning environments, applied to five OPF-Gym benchmark problems. Environment design decisions (reward terms, observation additions, training-data mixture, episode length, action scaling) are treated as hyperparameters and optimized with NSGA-III, using a short DDPG inner loop (40k steps, three seeds) and validation-data metrics for constraint satisfaction and objective error. The optimized designs are compared against a manually derived baseline from the authors' prior work. The paper reports that the automated designs outperform or match the baseline across the five benchmarks, uses statistical tests to identify which design decisions matter, and verifies two of the five environments with 500k-step training, held-out test data, and a second RL algorithm (SAC), noting a risk of environment overfitting to the RL algorithm.

Significance. If the central claim holds, the paper provides a practical and general methodology for RL environment design in power systems, with immediate actionable recommendations (Diff-Objective, Autoscaling, mixed training data) and an open-source implementation. The work is empirically grounded: it uses five open benchmarks, held-out test data for verification, longer training runs on two environments, a second RL algorithm, and reproducibility artifacts. The main caveat is that the load-bearing proxy assumption — that short DDPG-based optimization scores predict deployment performance — is verified on only a subset of the benchmarks and is partially contradicted by the SAC transfer result. With appropriate verification or qualification, the contribution would be a useful step toward automating environment design for RL-OPF.

major comments (4)
  1. [Abstract, Section 5.2, Section 8] The claim that automated designs "consistently outperform" the baseline on five OPF benchmarks is not fully supported by the presented evidence. Section 8 verifies the optimized designs at 500k steps, with the full training set and held-out test data, for only two of the five environments (Economic Dispatch and Voltage Control). For Load Shedding, Q-Market, and Max Renewables, the performance comparison rests entirely on the 40k-step, validation-data, three-seed inner-loop evaluation. Since the entire methodology depends on the proxy being predictive of deployment performance, the paper should either extend the verification to the remaining three environments or explicitly restrict the claim to the verified settings.
  2. [Section 8, Figure 6] The proxy is shown to be not algorithm-robust. For Economic Dispatch, the automated design combined with SAC achieves near-perfect constraint satisfaction but the worst optimization performance of the four combinations, which the authors themselves describe as "overadjustment of the environment to the RL algorithm, similar to overfitting." This is a concrete counterexample to the unqualified claim that the optimized environment design is generally superior, and it means the transferability of the design across RL algorithms cannot be assumed. The abstract's unqualified "consistently outperforms" should be qualified to "with the same RL algorithm," consistent with the more careful wording in Section 8.
  3. [Section 7, Figure 4] The statistical analysis in Section 7 tests 15 design variables against four evaluation criteria, resulting in 60 tests per environment, and additionally an aggregate analysis with Fisher's method, all using a per-test significance threshold of p < 0.05. No multiple-comparison correction is applied, so under the null hypothesis one would expect roughly three false positives per environment by chance. The "statistically significant design decisions" lists in Figures 4, 5, and Appendix B, which underlie several of the paper's novel insights, are therefore not reliable as reported. The authors should report corrected p-values (e.g., Benjamini-Hochberg or Bonferroni) or otherwise justify why multiple-comparison control is unnecessary here.
  4. [Section 4.2, Section 5.2, Table 1] The search space advertised as 15 variables is not actually searched as such. Table 1 restricts Steps Per Episode to {1} after "initial experiments," and the footnote in Section 4.2 states that seven additional implemented parameters were omitted because they "yielded no noteworthy results." Both decisions are based on undocumented pre-studies, and restricting a dimension after observing its effect on the validation metric is a form of peeking at the optimization target. This is not necessarily fatal, but the paper should disclose these reductions prominently in the experimental setup and avoid the unqualified statement that the methodology optimized over the full 15-variable space.
minor comments (5)
  1. [Section 4.2, Eq. (13)] The word "adapated" in the paragraph before Eq. (13) is a typo; it should be "adapted."
  2. [Section 5.2] The statement that NSGA-IIISampler "was chosen because it outperformed other optimizers in undocumented pre-studies" appears several times in the paper (also for DDPG and the 20/80 split). For reproducibility, these pre-studies should be documented or made available in the repository, at least in summary form.
  3. [Section 4.2, Eq. (12)] The normalization of the reward components is stated to ensure that performance differences can be attributed to function shape rather than magnitude, but the claim implicitly assumes that normalization does not change which environment design is best. This assumption should be stated explicitly and justified, as reward scale changes are known to interact with RL algorithm hyperparameters.
  4. [Section 6, Figures 2 and 3] The text says the automated design "significantly outperform[s]" the manual design for Economic Dispatch, but no statistical test accompanies the Pareto-front visualization. Since the manual design uses ten seeds and the automated samples use three, a direct statistical comparison of the relevant performance metrics would strengthen the claim.
  5. [Section 9] The sentence "In none of the five use cases, the manual design achieved performances below the Pareto-front of the automated design" is ambiguous: for Load Shedding, Figure 3 places the manual designs on the far right end of the Pareto front, which is consistent with being competitive rather than dominated. The wording should be clarified to say that the manual design never strictly dominates automated designs, or similar.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claim is an empirical comparison with held-out verification, not a derivation from fitted inputs.

full rationale

The paper's load-bearing claim is empirical: multi-objective HPO over a 15-dimensional environment-design space produces Pareto-front designs that dominate a manually specified baseline on validation metrics, and Section 8 re-evaluates the selected designs on held-out test data with 500k training steps, the full training set, and a second RL algorithm (SAC). None of the equations (e.g., the parameterized reward in Eqs. 12-14, the data-mixing sampler in Eq. 9, or the action-space variants in Eq. 16) defines the compared performance metrics in terms of the optimized design variables, and no parameter is fitted to the test data and then renamed as a prediction. The main self-citations are the OPF-Gym benchmarks [36] and the prior environment-design study [39] used as the manual baseline; these are domain artifacts and motivation, not load-bearing premises, and the comparison is executed in this paper with separate validation/test splits, multiple seeds, and four evaluation points. Section 8 explicitly flags that the 40k-step DDPG proxy is only verified on two of five benchmarks and that the automated design does not fully transfer to SAC ('overadjustment of the environment to the RL algorithm, similar to overfitting'); these are honest limitations on external validity and do not constitute circularity. No uniqueness theorem or ansatz is imported via citation to force the conclusion. Therefore no circular step is identified.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The ledger shows 8 hand-chosen or HPO-fitted numbers on which the central comparison rests: 4 reward parameters, 3 data-mixture shares, plus hand-made exclusions (Steps Per Episode restricted to 1, the 20/80 significance split) and the 40k-step proxy budget. The 15-variable search space also includes 5 observation booleans and 2 action/reward booleans that are categorical, not numbers, and are therefore not listed. The axioms are dominated by domain assumptions about benchmark correctness and representativeness; the ad hoc normalization and two-metric objective are inherited constructs. No invented entities: R_design is a recombination of reward variants from the cited literature. The 'novel insights' of Section 7 are descriptive statistics over the same runs that produced the fitted designs, so their independent status rests on the Section 8 verification.

free parameters (8)
  • Penalty Weight (beta) = 0.54 (Best Eco), 0.16 (Best Voltage)
    Reward parameter in [0.01, 0.99] weighting penalty vs objective in Eq. 12; searched by NSGA-II; significant for all four evaluation criteria across environments.
  • Invalid Objective Share (psi) = 0.80 (Best Eco), 0.47 (Best Voltage)
    Interpolates the invalid-state objective term in Eq. 13 between additive (psi=1) and replace-style (psi=0) rewards; HPO-fitted.
  • Valid Reward = 0.88 (Best Eco), 0.97 (Best Voltage)
    Offset for valid states in Eq. 14, searched in [0, 2.0]; the baseline [39] uses 0.0.
  • Invalid Penalty = 1.11 (Best Eco), 0.57 (Best Voltage)
    Offset for invalid states in Eq. 14, searched in [0, 2.0]; the baseline [39] uses 0.0.
  • Training data mixture shares = Normal 23.8/35.5%, Uniform 41.2/28.7%, Realistic 35.0/35.8% (Best Eco/Best Voltage)
    Probabilities in Eq. 9 constrained to sum to 1.0; HPO-fitted; a low Realistic share is associated with non-dominated solutions.
  • Steps Per Episode = 1 (restricted post hoc)
    Originally {1,3,5}, restricted to {1} after initial experiments because n-step variants performed drastically worse (Table 1); a hand-made exclusion that shapes the central comparison.
  • Top-20% split for significance tests = 20%
    Group split for Section 7 chosen by 'undocumented sensitivity analysis'; affects which design decisions appear statistically significant.
  • Optimization training budget = 40k steps (DDPG)
    Chosen for tractability; the proxy validity of short-run performance is verified for only two of five environments in Section 8.
assumptions (6)
  • domain assumption OPF-Gym provides correct ground-truth OPF optima and a valid baseline conventional solver for the two metrics.
    Section 4.1 defines invalid share omega and mean valid error DeltaJ against baseline solver solutions; if the baseline solver is wrong for some states, both metrics inherit the error, and the negative metric values (RL beating the solver) would be artifacts.
  • domain assumption The five OPF-Gym benchmark environments are representative enough to support general RL-OPF environment design conclusions.
    Section 9 concedes that multi-stage and stochastic OPF variants with sequential decisions are not covered, and that the 1-step result is 'most probably not transferable' to the multi-stage OPF, so the general conclusions only hold for single-step OPF use cases.
  • standard math Welch's t-test, the chi-squared test, and Fisher's method are valid for comparing top-20% and bottom-80% groups drawn from the same HPO runs.
    Section 7 relies on these tests. The 500 samples are not independent across HPO iterations because NSGA-II adaptively selects new candidates based on previous results, which the tests assume they are.
  • domain assumption Three-seed averaging during optimization and ten-seed averaging during verification sufficiently control RL stochasticity.
    Sections 5.2 and 8; the SAC verification curves in Figure 6 show high standard deviations and multiple outliers, indicating that residual stochasticity can be substantial.
  • ad hoc to paper Normalizing the reward to zero mean and unit variance is necessary for fair comparison and does not change which environment design is best.
    Section 4.2 normalizes the reward 'to mean of zero and a variance of one' and the baseline [39] design is modified to use normalized rewards; the assumption that magnitude alone does not alter relative rankings is stated, not derived.
  • domain assumption The two metrics (invalid share and mean valid error) capture the full relevant RL-OPF performance.
    Section 4.1 defines only these two objectives; properties such as worst-case failures, robustness to solver failures, and computational overhead are folded into or omitted from these two scalar metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A General Approach of Automated Environment Design for Learning the Optimal Power Flow." pith.science (2026). https://pith.science/paper/2Z5FRBVQ

@misc{pith2026250507832,
  author       = {Pith},
  title        = {Pith review of: A General Approach of Automated Environment Design for Learning the Optimal Power Flow},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2Z5FRBVQ}},
  note         = {Machine review of arXiv:2505.07832}
}
read the original abstract

Reinforcement learning (RL) algorithms are increasingly used to solve the optimal power flow (OPF) problem. Yet, the question of how to design RL environments to maximize training performance remains unanswered, both for the OPF and the general case. We propose a general approach for automated RL environment design by utilizing multi-objective optimization. For that, we use the hyperparameter optimization (HPO) framework, which allows the reuse of existing HPO algorithms and methods. On five OPF benchmark problems, we demonstrate that our automated design approach consistently outperforms a manually created baseline environment design. Further, we use statistical analyses to determine which environment design decisions are especially important for performance, resulting in multiple novel insights on how RL-OPF environments should be designed. Finally, we discuss the risk of overfitting the environment to the utilized RL algorithm. To the best of our knowledge, this is the first general approach for automated RL environment design.

Figures

Figures reproduced from arXiv: 2505.07832 by the authors.

Figure 1
Figure 1. Inner and outer loop of the multi-objective [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Pareto-front and distribution of samples for the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Pareto-front and distribution of samples for the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Statistically significant environment design deci [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Load Shedding Pareto-front and distribution of re￾sults, including all statistically significant environment de￾sign decisions. design because it performed well in trial runs, but it remains unclear if the environments that were optimized for DDPG also perform well wit…
Figure 6
Figure 6. Figure 6: Learning curves for both verification environments, [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 10
Figure 10. Figure 10: Economic Dispatch Pareto-front and distribution of results, including all statistically significant environment design decisions. B.5 Load Shedding Validity: Pen. Weight: 0.85 (high) Realistic Data: 0.21 (low) Uniform Data: 0.45 (high) Autoscale Act: False Optimizatio…
Figure 11
Figure 11. Figure 11: Load Shedding Pareto-front and distribution of results, including all statistically significant environment design decisions. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 9
Figure 9. Figure 9: Voltage Control Pareto-front and distribution of results, including all statistically significant environment design decisions. B.4 Economic Dispatch Validity: Pen. Weight: 0.62 (high) Invalid Obj.: 0.39 (low) Diff-Objective: True Optimization: Penalty Weight: 0.39 (lo…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 26 canonical work pages

  1. [36]

    Thomas Wolgast. 2024. OPF-Gym. https://github.com/Digitalized-Energy- Systems/opfgym

    work page 2024

  2. [39]

    Thomas Wolgast and Astrid Nieße. 2024. Learning the optimal power flow: Environment design matters. Energy and AI 17 (2024), 100410. https://doi.org/ 10.1016/j.egyai.2024.100410

    arXiv 2024

  3. [1]

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Frame- work. In Proceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining (KDD ’19) . Association for Computing Machinery, New York, NY, USA, 2623–2631. https://doi.org/10.1145...

    arXiv 2019

  4. [2]

    Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne-Laure Boulesteix, Difan Deng, and Marius Lindauer. 2023. Hyperparameter Optimization: Founda- tions, Algorithms, Best Practices, and Open Challenges. WIREs Data Mining and Knowledge Discovery 13, 2 (2023), e1484. https://d...

    doi:10.1002/widm.1484 2023

  5. [3]

    Florin Capitanescu. 2016. Critical Review of Recent Advances and Further Devel- opments Needed in AC Optimal Power Flow. Electric Power Systems Research 136 (July 2016), 57–68. https://doi.org/10.1016/j.epsr.2016.02.008

    doi:10.1016/j.epsr.2016.02.008 2016

  6. [4]

    Kalyanmoy Deb and Himanshu Jain. 2014. An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I: Solving Problems With Box Constraints. IEEE Transactions on Evolutionary Computation 18, 4 (Aug. 2014), 577–601. https://doi.org/10.1109/ TEVC.2013.2281535

    arXiv 2014

  7. [5]

    Theresa Eimer, Marius Lindauer, and Roberta Raileanu. 2023. Hyperparame- ters in Reinforcement Learning and How To Tune Them. In Proceedings of the 40th International Conference on Machine Learning . PMLR, 9104–9149. https: //proceedings.mlr.press/v202/eimer23a.html

    work page 2023

  8. [6]

    Timm Faulwasser, Alexander Engelmann, Tillmann Mühlpfordt, and Veit Ha- genmeyer. 2018. Optimal power flow: an introduction to predictive, distributed and stochastic control challenges. at - Automatisierungstechnik 66, 7 (July 2018), 573–589. https://doi.org/10.1515/auto-2018-0040

    work page doi:10.1515/auto-2018-0040 2018

Show all 58 references
  1. [7]

    R. A. Fisher. 1992. Statistical Methods for Research Workers. In Breakthroughs in Statistics: Methodology and Distribution , Samuel Kotz and Norman L. Johnson (Eds.). Springer, New York, NY, 66–70. https://doi.org/10.1007/978-1-4612-4380- 9_6

    1992 doi

  2. [8]

    Stephen Frank, Ingrida Steponavice, and Steffen Rebennack. 2012. Optimal Power Flow: A Bibliographic Survey I. Energy Systems 3, 3 (Sept. 2012), 221–258. https://doi.org/10.1007/s12667-012-0056-y

    2012 doi

  3. [9]

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning. PMLR, 1861–1870. https://proceeding...

    2018

  4. [10]

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep Reinforcement Learning That Matters. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Inno- vative Applications of Artificial In...

    2018

  5. [11]

    Wanjun Huang, Minghua Chen, and Steven H. Low. 2024. Unsupervised Learning for Solving AC Optimal Power Flows: Design, Analysis, and Experiment. IEEE Transactions on Power Systems (2024), 1–13. https://doi.org/10.1109/TPWRS.2024. 3373399

    2024 doi

  6. [12]

    Anssi Kanervisto, Christian Scheller, and Ville Hautamäki. 2020. Action Space Shaping in Deep Reinforcement Learning. In 2020 IEEE Conference on Games (CoG). 479–486. https://doi.org/10.1109/CoG47356.2020.9231687

    2020

  7. [13]

    Hooman Khaloie, Mihály Dolányi, Jean-François Toubeau, and François Vallée

  8. [14]

    Joanne Taery Kim and Sehoon Ha. 2021. Observation Space Matters: Benchmark and Optimization Algorithm. In2021 IEEE International Conference on Robotics and Automation (ICRA). 1527–1534. https://doi.org/10.1109/ICRA48506.2021.9561019

    2021

  9. [15]

    James Kotary, Ferdinando Fioretto, Pascal Van Hentenryck, and Bryan Wilder

  10. [16]

    Lillicrap, Jonathan J

    Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016. Continuous Control with Deep Reinforcement Learning. https://doi.org/10.48550/arXiv.1509.02971 arXiv:1509.02971 [cs, stat]

    2016 doi arXiv

  11. [17]

    Chenyuchuan Liu, Yan Li, and Tianqi Xu. 2024. A Neural Network Approach to Physical Information Embedding for Optimal Power Flow. Sustainability 16, 17 (Jan. 2024), 7498. https://doi.org/10.3390/su16177498

    2024 doi

  12. [18]

    Xinghua Liu, Bangji Fan, and Jiaqiang Tian. 2022. Deep Reinforcement Learn- ing Based Approach for Dynamic Optimal Power Flow in Active Distribution Network. In 2022 41st Chinese Control Conference (CCC) . 1951–1956. https: //doi.org/10.23919/CCC55666.2022.9902611

    2022

  13. [19]

    Maryam Majzoubi, Chicheng Zhang, Rajan Chari, Akshay Krishnamurthy, John Langford, and Aleksandrs Slivkins. 2020. Efficient Contextual Bandits with Continuous Actions. InAdvances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 349–360. https://proce...

    2020

  14. [20]

    Ng, Daishi Harada, and Stuart J

    Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. 1999. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the Sixteenth International Conference on Machine Learning (ICML ’99). Morgan Kaufmann Publishers Inc., San Fra...

    1999

  15. [21]

    Huanhuan Nie, Ying Chen, Yankan Song, and Shaowei Huang. 2019. A General Real-time OPF Algorithm Using DDPG with Multiple Simulation Platforms. In 2019 IEEE Innovative Smart Grid Technologies - Asia (ISGT Asia) . 3713–3718. https://doi.org/10.1109/ISGT-Asia.2019.8881174

    2019

  16. [22]

    Jingping Nie, Yanchen Liu, Liwei Zhou, Xiaofan Jiang, and Matthias Preindl

  17. [23]

    Damian Owerko, Fernando Gama, and Alejandro Ribeiro. 2024. Unsupervised Optimal Power Flow Using Graph Neural Networks. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 6885–6889. https://doi.org/10.1109/ICASSP48485.2024...

    2024

  18. [24]

    Xiang Pan, Tianyu Zhao, Minghua Chen, and Shengyu Zhang. 2021. DeepOPF: A Deep Neural Network Approach for Security-Constrained DC Optimal Power Flow. IEEE Transactions on Power Systems 36, 3 (May 2021), 1725–1735. https: //doi.org/10.1109/TPWRS.2020.3026379

    2021

  19. [25]

    Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. 2018. Time Limits in Reinforcement Learning. In Proceedings of the 35th International Confer- ence on Machine Learning . PMLR, 4045–4054. https://proceedings.mlr.press/v80/ pardo18a.html

    2018

  20. [26]

    Mak, and Pascal Van Hentenryck

    Seonho Park, Wenbo Chen, Terrence W.K. Mak, and Pascal Van Hentenryck. 2024. Compact Optimization Learning for AC Optimal Power Flow. IEEE Transactions on Power Systems 39, 2 (March 2024), 4350–4359. https://doi.org/10.1109/TPWRS. 2023.3313438

    2024

  21. [27]

    Karl Pearson. 1900. X. On the Criterion That a given System of Deviations from the Probable in the Case of a Correlated System of Variables Is Such That It Can Be Reasonably Supposed to Have Arisen from Random Sampling . The London, Edinburgh, and Dublin Philosophical Magazine...

    1900 doi

  22. [28]

    Xue Bin Peng and Michiel van de Panne. 2017. Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?. In Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation (SCA ’17) . Association for Computing Machinery, New York, NY, USA, ...

    2017

  23. [29]

    Daniele Reda, Tianxin Tao, and Michiel van de Panne. 2020. Learning to Loco- mote: Understanding How Environment Design Matters for Deep Reinforcement Learning. In Proceedings of the 13th ACM SIGGRAPH Conference on Motion, Inter- action and Games (MIG ’20) . Association for Co...

    2020

  24. [30]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  25. [31]

    Henry Sowerby, Zhiyuan Zhou, and Michael L. Littman. 2022. Design- ing Rewards for Fast Learning. https://doi.org/10.48550/arXiv.2205.15400 arXiv:2205.15400 [cs]

    2022 doi arXiv

  26. [32]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Intro- duction (second edition ed.). The MIT Press, Cambridge, Massachusetts

    2018

  27. [33]

    Hado P van Hasselt, Arthur Guez, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver. 2016. Learning Values across Many Orders of Magnitude. In Advances in Neural Information Processing Systems , Vol. 29. Curran Associates, Inc. https://proceedings.neurips.cc/paper/20...

    2016

  28. [34]

    Junfei Wang and Pirathayini Srikantha. 2023. Fast Optimal Power Flow With Guarantees via an Unsupervised Generative Model. IEEE Transactions on Power Systems 38, 5 (Sept. 2023), 4593–4604. https://doi.org/10.1109/TPWRS.2022. 3212925

    2023 doi

  29. [35]

    Bernard L. Welch. 1947. The Generalization of ‘STUDENT’S’Problem When Several Different Population Varlances Are Involved. Biometrika 34, 1-2 (1947), 28–35. https://academic.oup.com/biomet/article-pdf/34/1--2/28/553093/34--1-- 2--28.pdf

    1947

  30. [37]

    Thomas Wolgast. 2025. Whitepaper: Environment Design for Reinforcement Learning: A Practical Guide and Overview. ResearchGate (2025). https://doi.org/ 10.13140/RG.2.2.28673.77925

    2025

  31. [38]

    Thomas Wolgast and Astrid Nieße. 2024. Approximating Energy Market Clearing and Bidding With Model-Based Reinforcement Learning. IEEE Access 12 (2024), 145106–145117. https://doi.org/10.1109/ACCESS.2024.3472480

    2024

  32. [40]

    Thomas Wolgast, Eric MSP Veith, and Astrid Nieße. 2021. Towards Reinforce- ment Learning for Vulnerability Analysis in Power-Economic Systems. Energy Informatics 4, 3 (Sept. 2021), 21. https://doi.org/10.1186/s42162-021-00181-5

    2021 doi

  33. [41]

    Jong Ha Woo, Lei Wu, Jong-Bae Park, and Jae Hyung Roh. 2020. Real-Time Optimal Power Flow Using Twin Delayed Deep Deterministic Policy Gradient Algorithm. IEEE Access 8 (2020), 213611–213618. https://doi.org/10.1109/ACCESS. 2020.3041007

    2020

  34. [42]

    Tong Wu, Anna Scaglione, and Daniel Arnold. 2024. Constrained Reinforcement Learning for Predictive Control in Real-Time Stochastic Dynamic Optimal Power Flow. IEEE Transactions on Power Systems 39, 3 (May 2024), 5077–5090. https: //doi.org/10.1109/TPWRS.2023.3326121

    2024

  35. [43]

    Yizhi Wu, Yujian Ye, Jianxiong Hu, Peilin Zhao, Liu Liu, Goran Strbac, and Chongqing Kang. 2024. Chance Constrained MDP Formulation and Bayesian Advantage Policy Optimization for Stochastic Dynamic Optimal Power Flow. IEEE Transactions on Power Systems 39, 5 (Sept. 2024), 6788...

    2024

  36. [44]

    Ziming Yan and Yan Xu. 2020. Real-Time Optimal Power Flow: A Lagrangian Based Deep Reinforcement Learning Approach. IEEE Transactions on Power Sys- tems 35, 4 (July 2020), 3270–3273. https://doi.org/10.1109/TPWRS.2020.2987292

    2020

  37. [45]

    Zhongkai Yi, Xue Wang, Cheng Yang, Chao Yang, Mengyang Niu, and Wotao Yin

  38. [46]

    Zamzam and Kyri Baker

    Ahmed S. Zamzam and Kyri Baker. 2020. Learning Optimal Solutions for Ex- tremely Fast AC Optimal Power Flow. In 2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (Smart- GridComm). 1–6. https://doi.org/10.1109/SmartGridCo...

    2020

  39. [47]

    Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. 2018. A Study on Overfitting in Deep Reinforcement Learning. https://doi.org/10.48550/arXiv. 1804.06893 arXiv:1804.06893 [cs, stat]

    2018 doi arXiv

  40. [48]

    Linrui Zhang, Qin Zhang, Li Shen, Bo Yuan, Xueqian Wang, and Dacheng Tao

  41. [49]

    Hongyue Zhen, Zhai Hefeng, Ma Weizhe, Ligang Zhao, Weng Yixuan, Xu Yuan, Shi Jun, and He Xiaofeng. 2021. Design and Tests of Reinforcement-Learning- Based Optimal Power Flow Solution Generator. Energy Reports (2021). https: //doi.org/10.1016/j.egyr.2021.11.126

    2021 doi

  42. [50]

    Min Zhou, Minghua Chen, and Steven H. Low. 2023. DeepOPF-FT: One Deep Neural Network for Multiple AC-OPF Problems With Flexible Topology. IEEE Transactions on Power Systems 38, 1 (Jan. 2023), 964–967. https://doi.org/10.1109/ TPWRS.2022.3217407

    2023

  43. [51]

    Yuhao Zhou, Wei-Jen Lee, Ruisheng Diao, and Di Shi. 2022. Deep Reinforcement Learning Based Real-time AC Optimal Power Flow Considering Uncertainties. Journal of Modern Power Systems and Clean Energy 10, 5 (Sept. 2022), 1098–1109. https://doi.org/10.35833/MPCE.2020.000885

    2022

  44. [52]

    Yuhao Zhou, Bei Zhang, Chunlei Xu, Tu Lan, Ruisheng Diao, Di Shi, Zhiwei Wang, and Wei-Jen Lee. 2020. A Data-driven Method for Fast AC Optimal Power Flow Solutions via Deep Reinforcement Learning. Journal of Modern Power Systems and Clean Energy 8, 6 (Nov. 2020), 1128–1139. ht...

    2020

  45. [2017]

    https://doi.org/10.48550/arXiv

    Proximal Policy Optimization Algorithms. https://doi.org/10.48550/arXiv. 1707.06347 arXiv:1707.06347 [cs]

    doi arXiv

  46. [2021]

    https://doi

    End-to-End Constrained Optimization Learning: A Survey. https://doi. org/10.48550/arXiv.2103.16378 arXiv:2103.16378 [cs]

    doi arXiv

  47. [2022]

    In 2022 IEEE Transportation Electrification Conference & Expo (ITEC)

    Deep Reinforcement Learning Based Approach for Optimal Power Flow of Microgrid with Grid Services Implementation. In 2022 IEEE Transportation Electrification Conference & Expo (ITEC) . IEEE, Anaheim, CA, USA, 1148–1153. https://doi.org/10.1109/ITEC53557.2022.9813862

    2022

  48. [2023]

    Proceedings of the AAAI Conference on Artificial Intelligence 37, 12 (June 2023), 15313–15321

    Evaluating Model-Free Reinforcement Learning toward Safety-Critical Tasks. Proceedings of the AAAI Conference on Artificial Intelligence 37, 12 (June 2023), 15313–15321. https://doi.org/10.1609/aaai.v37i12.26786

    2023 doi

  49. [2024]

    IEEE Transactions on Power Systems 39, 1 (Jan

    Real-Time Sequential Security-Constrained Optimal Power Flow: A Hybrid Knowledge-Data-Driven Reinforcement Learning Approach. IEEE Transactions on Power Systems 39, 1 (Jan. 2024), 1664–1680. https://doi.org/10.1109/TPWRS. 2023.3262843

    2024

  50. [2025]

    Applied Energy 388 (2025), 125637

    Review of machine learning techniques for optimal power flow. Applied Energy 388 (2025), 125637. https://doi.org/10.1016/j.apenergy.2025.125637

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.