Comparative Analysis of Machine Learning Models for Short-Term Distribution System Load Forecasting

Cunzhi Zhao; Elias Raffoul; Meng Ling; Mingjian Tuo; Tianxia Zhao; Xingpeng Li

REVIEW 4 major objections 5 minor 16 references

Comparative Analysis of Machine Learning Models for Short-Term Distribution System Load Forecasting

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper tests five machine learning models for hour-ahead load forecasting on a synthetic Houston distribution system and finds that recurrent and feedforward networks give the lowest error, while the attention graph network lags.

desk verdict Useful benchmark of five ML forecasters on a synthetic Houston feeder, but the missing train/test split undermines the headline ranking. read the letter →

arxiv 2411.16118 v1 pith:R4IXVVTT submitted 2024-11-25 eess.SY cs.SY

Elias Raffoul , Mingjian Tuo , Cunzhi Zhao , Tianxia Zhao , Meng Ling , Xingpeng Li This is my paper · ORCID

classification eess.SYcs.SY

keywords loadforecastingdistributionsystemrecurrentneuralnetworkfeedforwardlongshort-termmemorygatedunitattentiontemporalgraphconvolutionalsyntheticdata

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper compares five machine learning approaches for predicting the next hour of electrical load on a 44-node distribution system, using a 24-hour look-back window and training on one or five years of hourly data. Its central finding is that recurrent neural networks and feedforward neural networks produce the lowest forecast errors, while the attention temporal graph convolutional network, which explicitly models spatial and temporal structure, produces the largest errors. The authors interpret this as evidence that added architectural complexity does not automatically help for short-term forecasting on this kind of system. The result is specific to the synthetic Energy Corridor feeder, but it offers a caution for practitioners who might assume that more sophisticated spatiotemporal models are always better.

What carries the argument

The evaluation rests on a controlled comparison: every model sees the same input format, a tensor of shape [batch, 24, 88] covering active and reactive power at all 44 buses, and is trained for 100 epochs with the same three error metrics, MAE, MSE, and MAPE. The model family ranges from dense feedforward networks to gated recurrent units to A3T-GCN, which couples graph convolutional layers with GRUs and an attention mechanism to combine spatial and temporal information. The synthetic dataset itself is a second piece of machinery: residential and commercial load shapes are simulated with building energy software and scaled to Houston-style consumption levels, then formatted into hourly nodal P and Q values.

What would settle it

Train the same five models with the same look-back and metrics on real measured distribution feeder data—for instance, one to five years of hourly or 15-minute smart meter records—and compare MAE, MSE, and MAPE across models; a different model ordering would show that the paper's conclusion is specific to its synthetic data.

Watch

Extended reading notes

Core claim

On the synthetic Energy Corridor feeder, the paper reports that FNN and RNN are the top performers in hour-ahead load forecasting. For the one-year dataset, FNN has the lowest MAE (26.94) and MSE (2510.29), while RNN has the lowest MAPE (25.58). For the five-year dataset, RNN records the lowest MAE (4.70), MSE (80.47), and MAPE (21.05), with FNN a close second. LSTM and GRU perform in the middle, and A3T-GCN is the least accurate on every metric in both settings. The paper attributes A3T-GCN's poor performance to overfitting and high computational demand on datasets that are not large or diverse enough to exploit its spatiotemporal capacity.

Load-bearing premise

The rankings depend on the synthetic load dataset behaving like a real feeder; if real loads are noisier or more spatially correlated, the relative performance of the models, especially A3T-GCN, could change.

Editorial extensions

If this is right

Operators of similar distribution feeders can use FNN or RNN for hour-ahead forecasts and expect competitive accuracy at a fraction of the training cost.
The A3T-GCN result warns against adopting spatiotemporal graph models without running simpler baselines first.
More training data (five years vs one year) improves every model, so data collection matters as much as architecture selection.
The large per-epoch training time gap makes simpler models more practical for frequent retraining in operational settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The one-hour-ahead horizon is short; at day-ahead horizons the spatial correlations captured by A3T-GCN could matter more, and the model ranking may reverse.
Synthetic profiles are usually smoother and less spatially coupled than real feeder measurements, so the FNN/RNN advantage should shrink or disappear on measured data.
A direct test would be to rerun the same protocol on real feeder data; if A3T-GCN climbs in the rankings, the paper's conclusion is an artifact of the synthetic dataset.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper does something genuinely useful: it compares FNN, RNN, LSTM, GRU, and A3T-GCN for one-hour-ahead load forecasting on a synthetic 44-bus distribution feeder, and it grounds the synthetic data in standard EnergyPlus and ASHRAE prototype models tied to Houston demographics. That is externally reproducible in spirit, and the comparison across 1-year and 5-year datasets is a reasonable setup. The main empirical finding—that simpler RNN/FNN models beat the graph-convolutional A3T-GCN—is plausible and consistent with some prior work, and the paper is honest about the computational cost of A3T-GCN (16 minutes per epoch vs. 40 seconds). I give credit for reporting MAE, MSE, MAPE, and tolerance curves rather than just one metric.

The soft spot is load-bearing: the paper never describes a train/validation/test split. Section IV says only that 'following training over 100 epochs, each model was evaluated' and Table 1 appears to report metrics on that same training data. Without a held-out chronological split, the numbers are in-sample fits, not one-hour-ahead forecasts. The authors' own overfitting explanation for A3T-GCN is internally inconsistent—you cannot detect overfitting without a held-out set. This is a bigger problem than the synthetic-data concern; even on simulated profiles, a proper split would make the comparison meaningful. Missing hyperparameters (learning rate, hidden sizes, batch size) and no error bars or repeated-seed variance further prevent assessing robustness. The A3T-GCN MAPE of 85.28% is so far above the others that it suggests scaling or training issues rather than a reliable architectural comparison.

The synthetic dataset limitation is real but secondary. The profiles are not validated against measured feeder data, and the relative performance of graph-based models could shift with more realistic spatial correlation. Still, the models are standard, the data generation is grounded in public standards, and the paper does not oversell its conclusions—the conclusion explicitly notes the results may vary for other systems.

This paper deserves a serious referee, but it needs major revision: define the data split, report uncertainty, and provide hyperparameters or code. As written, the central ranking is unverifiable. I would not cite it in its current form, but I would be curious to see a revised version.

Recommendation: send to peer review with a clear request for a chronological split and reproducibility details.

Referee Report

4 major / 5 minor

Summary. The paper compares five machine learning models (FNN, RNN, LSTM, GRU, and A3T-GCN) for one-hour-ahead load forecasting on a synthetic 44-bus distribution network representing Houston's Energy Corridor. The input is an 88-dimensional vector of active and reactive loads at all buses with a 24-hour look-back window. Models are trained on one-year and five-year synthetic datasets and evaluated with MAE, MSE, and MAPE. The authors report that RNN and FNN achieve the lowest errors, while A3T-GCN performs worst, and attribute this to overfitting and the higher complexity of A3T-GCN. The conclusion recommends simpler models for this task and suggests that A3T-GCN may be better suited to longer horizons in future work.

Significance. If the comparison were rigorously controlled, this would be a useful benchmark for distribution-level short-term load forecasting on publicly reproducible synthetic data. The paper's strengths include the use of standard building simulation tools (EnergyPlus, ASHRAE prototypes, TMY3 weather) to generate the dataset, which gives some transparency to the data-generation process. However, the central claim (that RNN/FNN outperform A3T-GCN) is not verifiable as reported because the evaluation protocol is unspecified, and the dataset is described inconsistently as both real and synthetic. The paper also ships no code or hyperparameter details, so the empirical contribution is limited. The reported results are plausible but currently unsubstantiated.

major comments (4)

[Section IV / Table 1] The manuscript never specifies a train/validation/test split. Section IV states only that "following training over 100 epochs, each model was evaluated" and reports a single MAE/MSE/MAPE per model. With 8,760 or 43,800 hourly samples, if the metrics are computed on the training data, they reflect in-sample fit rather than one-hour-ahead generalization. This directly undermines the paper's central conclusion that RNN and FNN are the most accurate models. A proper chronological split (e.g., last year or last 20% of the five-year series held out) and a description of validation-based hyperparameter selection are required before the ranking can be interpreted.
[Section I vs. Section III] The Introduction claims the study uses "real-world datasets from the 44-bus Energy Corridor distribution system," while Section III describes a "Synthetic Energy Corridor Distribution Network" with load profiles generated from EnergyPlus and ASHRAE prototype models. This contradiction is load-bearing because the external validity of the entire comparison depends on whether the data are measured or simulated. The authors must correct the Introduction and explicitly discuss the limitations of synthetic data, including the absence of validation against measured feeder data.
[Table 1, A3T-GCN rows] The A3T-GCN MAPE values (85.28% on one-year data, 84.44% on five-year data) are far outside the range of the other models (21–32%). A gap of this magnitude is more consistent with a scaling, implementation, or training-instability issue than with a genuine property of the architecture. The paper's explanation of overfitting is speculative without a held-out set and without training/validation curves beyond the accuracy-tolerance plots in Fig. 6. The authors should verify the A3T-GCN implementation, report its hyperparameters, and show per-epoch training and validation loss to demonstrate that the model was trained to convergence and is not misconfigured.
[Table 1 / Section IV] Each metric is reported as a single number with no variance across independent runs, no random seeds, and no hyperparameter details (hidden layer sizes, number of units, learning rate, batch size, optimizer, regularization). With 100 epochs and no error bars, the reported ranking may be within run-to-run noise. The authors should provide mean and standard deviation over multiple seeds, or at least state that the results are deterministic and why.

minor comments (5)

[Section V] The conclusion states that "RNN & FNN demonstrate the highest accuracy in short-term forecasting, sharing the lowest error values across datasets," but Table 1 shows that on the one-year dataset FNN has lower MAE and MSE while RNN has lower MAPE, and on the five-year dataset RNN is lower on all three metrics. The phrase "sharing" is imprecise and should be replaced with a per-metric statement.
[Figure 6] The figure caption and text refer to accuracy at error tolerances of 10%, 15%, and 20%, but the axes are not labeled clearly enough to determine whether the curves show the fraction of predictions within each tolerance. The text also mentions accuracy gains during the first 20–30 epochs, but no epoch axis is visible in the figure. Clarify the axis labels and the definition of "accuracy."
[Reference [14]] Reference [14] cites a blog (buildingsiot.com) rather than the original DOE/EIA source. The authors should cite the authoritative EIA Commercial Buildings Energy Consumption Survey (CBECS) or the DOE prototype building documentation.
[Section IV] The computational cost comparison is anecdotal ("about 40 seconds per epoch" versus "up to 16 minutes per epoch") and does not specify the hardware or software environment. Please provide the exact timing setup or remove the claim.
[General] The paper uses "1 & 5 YEARS DATASETS" in the Table 1 caption; this should be written as "1- and 5-Year Datasets" for grammatical clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the study is an empirical benchmark on externally generated synthetic data with no self-referential derivation.

full rationale

The paper contains no derivation chain that reduces to its own inputs. It benchmarks five standard machine learning architectures on a synthetic distribution system dataset generated from external building-simulation standards (EnergyPlus, ASHRAE prototype models, TMY3 data), not from the forecasting models themselves. The models are standard published architectures, and the A3T-GCN implementation is cited to external work. The self-citations in the introduction (references [2]–[5]) are illustrative examples of ML applications and are not load-bearing for the comparison. No parameter is fitted to a target and then renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The main methodological concern—that Section IV does not describe an explicit train/test split—is a reproducibility and validity issue, not a circularity: the paper does not state that evaluation occurred on the training data, and one cannot exhibit an equation or construction forcing the reported errors to equal the training fit. Because the central results rest on an external benchmark rather than on a self-referential argument, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim is a benchmark comparison. It relies on the synthetic data generation pipeline being realistic and on the unspecified model training choices. There are no new theoretical entities, but the model hyperparameters and dataset scaling factors are free parameters that affect the results and are not reported in sufficient detail.

free parameters (2)

Model hyperparameters (hidden layers, units, learning rate, batch size) = unknown
The paper does not report hyperparameter values, yet the relative performance of the models likely depends on them. Results are presented as is without sensitivity analysis.
Houston scaling percentages for commercial and residential mix = unknown
Section III scales base EnergyPlus profiles according to 'Houston's specific percentage profiles.' The exact percentages are not listed, and they determine the synthetic load shapes.

assumptions (3)

domain assumption EnergyPlus and ASHRAE prototype simulations produce realistic load profiles for the Houston area.
Section III uses these simulations without validating against measured feeder data, so the synthetic dataset is assumed to be representative.
domain assumption Average consumption values (37 kWh/day residential, 22.5 kWh/sq ft/year commercial) are representative for the Energy Corridor.
The paper adopts these values from EIA and DOE sources to set base load magnitudes; if they are unrepresentative, the absolute load levels and model comparisons change.
domain assumption The 44-bus network topology and load placement adequately capture the spatial dependencies of the real feeder.
The spatial graph structure is used by A3T-GCN. The synthetic topology is described but not compared with the actual Energy Corridor system, so the graph-based model may be evaluated on an unrealistic graph.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Comparative Analysis of Machine Learning Models for Short-Term Distribution System Load Forecasting." pith.science (2026). https://pith.science/paper/R4IXVVTT

@misc{pith2026241116118,
  author       = {Pith},
  title        = {Pith review of: Comparative Analysis of Machine Learning Models for Short-Term Distribution System Load Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R4IXVVTT}},
  note         = {Machine review of arXiv:2411.16118}
}

read the original abstract

Accurate electrical load forecasting is crucial for optimizing power system operations, planning, and management. As power systems become increasingly complex, traditional forecasting methods may fail to capture the intricate patterns and dependencies within load data. Machine learning (ML) techniques have emerged as powerful alternatives, offering superior prediction accuracy and the ability to model non-linear and complex temporal relationships. This study presents a comprehensive comparison of prominent ML models: feedforward neural networks, recurrent neural networks, long short-term memory networks, gated recurrent units, and the attention temporal graph convolutional network; for short-term load forecasting of the Energy Corridor distribution system in Houston, Texas. Using a 24-hour look-back window, we train the models on datasets spanning one and five years, to predict the load demand for the next hour and assess performance. Our findings aim to identify the most effective ML approach for accurate load forecasting, contributing to improved grid reliability and system optimization.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 11 canonical work pages

[1]

Artificial Intelligence for Load Forecasting,

T. Hong and P. Wang, “Artificial Intelligence for Load Forecasting,” IEEE Power & Energy Magazine, May/June 2022, doi: 10.1109/MPE.2022.3150808

arXiv 2022
[2]

Neural Network-based Power Flow Model,

T. Pham and X. Li, "Neural Network-based Power Flow Model," 2022 IEEE Green Technologies Conference (GreenTech), Houston, TX, USA, 2022, pp. 105-109, doi: 10.1109/GreenTech52845.2022.9772026

work page arXiv 2022
[3]

Wholesale Electricity Price Forecasting using Integrated Long-term Recurrent Convolutional Network Model

Vasudharini Sridharan, Mingjian Tuo and Xingpeng Li, “Wholesale Electricity Price Forecasting using Integrated Long-term Recurrent Convolutional Network Model”, Energies, 15(20), 7606, Oct. 2022

work page 2022
[4]

Microgrid Optimal Energy Scheduling Considering Neural Network based Battery Degradation

Cunzhi Zhao and Xingpeng Li, “Microgrid Optimal Energy Scheduling Considering Neural Network based Battery Degradation”, IEEE Transactions on Power Systems, early access, Jan. 2023

work page 2023
[5]

An Alternative Method for Solving Security-Constraint Unit Commitment with Neural Network Based Battery Degradation Model

Cunzhi Zhao and Xingpeng Li, “An Alternative Method for Solving Security-Constraint Unit Commitment with Neural Network Based Battery Degradation Model”, 54th North American Power Symposium, Salt Lake City, UT, USA, Oct. 2022

work page 2022
[6]

Power load forecasting in the spring festival based on feedforward neural network model,

R. ZhiChao, Y. Qiang, W. Haiyan, C. Chao and L. Yuan, "Power load forecasting in the spring festival based on feedforward neural network model," 2017 3rd IEEE International Conference on Computer and Communications (ICCC), Chengdu, China, 2017, pp. 2855-2858, doi: 10.1109/CompComm.2017.8323053

work page arXiv 2017
[7]

A Comparative Analysis of Deep Learning Models for Short-Term Load Forecasting,

I. Ul Haq, M. Khalid and U. Manzoor, "A Comparative Analysis of Deep Learning Models for Short-Term Load Forecasting," 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), Auckland, New Zealand, 2023, pp. 1-7, doi: 10.1109/CASE56687.2023.10260626

arXiv 2023
[8]

Comparison of Deep Learning-Based Methods for Electrical Load Forecasting

S., Angel T. et al. “Comparison of Deep Learning-Based Methods for Electrical Load Forecasting.” 2022 Third International Conference on Intelligent Computing Instrumentation and Control Technologies (ICICICT) (2022): 1598-1602

work page 2022

Show all 16 references

[9]

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

J. Chung, C. Gulcehre, K. Cho, Y. Bengio, “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling”, NIPS 2014 Deep Learning and Representation Learning Workshop, 2014, https://doi.org/10.48550/arXiv.1412.3555

2014 doi arXiv
[10]

A Comparative Study of LSTM/GRU Models for Energy Long-Term Forecasting in IoT Networks,

G. Goui, A. Zrelli and N. Benletaief, "A Comparative Study of LSTM/GRU Models for Energy Long-Term Forecasting in IoT Networks," 2023 IEEE/ACIS 23rd International Conference on Computer and Information Science (ICIS), Wuxi, China, 2023, pp. 60- 64, doi: 10.1109/ICIS57766.2023.10210257

2023
[11]

A3T-GCN: Attention Temporal Graph Convolutional Network for Traffic Forecasting

Zhu, J., Song, Y., Zhao, L., & Li, H, " A3T-GCN: Attention Temporal Graph Convolutional Network for Traffic Forecasting" 2020, Changsha, China https://arxiv.org/pdf/2006.11583.pdf

2020 arXiv
[12]

Short-Term Load Forecasting Using Recurrent Neural Networks With Input Attention Mechanism and Hidden Connection Mechanism,

M. Zhang, Z. Yu and Z. Xu, "Short-Term Load Forecasting Using Recurrent Neural Networks With Input Attention Mechanism and Hidden Connection Mechanism," in IEEE Access, vol. 8, pp. 186514- 186529, 2020, doi: 10.1109/ACCESS.2020.3029224

2020
[13]

Household Energy Use in Texas

U.S. Energy Information Administration, “Household Energy Use in Texas”, [Available Online]: https://www.eia.gov/consumption/residential/reports/2009/state_briefs /pdf/TX.pdf

2009
[14]

Building Energy Consumption Breakdown for Owners and Management

U.S. Energy Information Administration, “Building Energy Consumption Breakdown for Owners and Management”, 2023, [Available Online]: https://www.buildingsiot.com/blog/building- energy-consumption-breakdown-for-owners-and-management- bd#:~:text=According%20to%20Department%20of%...

2023
[15]

Commercial and Residential Hourly Load Profiles for all TMY3 Locations in the United States

Open Energy Data Initiative (OEDI), “Commercial and Residential Hourly Load Profiles for all TMY3 Locations in the United States”,
[2022]

[Available Online]: https://data.openei.org/submissions/153

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Artificial Intelligence for Load Forecasting,

T. Hong and P. Wang, “Artificial Intelligence for Load Forecasting,” IEEE Power & Energy Magazine, May/June 2022, doi: 10.1109/MPE.2022.3150808

arXiv 2022

[2] [2]

Neural Network-based Power Flow Model,

T. Pham and X. Li, "Neural Network-based Power Flow Model," 2022 IEEE Green Technologies Conference (GreenTech), Houston, TX, USA, 2022, pp. 105-109, doi: 10.1109/GreenTech52845.2022.9772026

work page arXiv 2022

[3] [3]

Wholesale Electricity Price Forecasting using Integrated Long-term Recurrent Convolutional Network Model

Vasudharini Sridharan, Mingjian Tuo and Xingpeng Li, “Wholesale Electricity Price Forecasting using Integrated Long-term Recurrent Convolutional Network Model”, Energies, 15(20), 7606, Oct. 2022

work page 2022

[4] [4]

Microgrid Optimal Energy Scheduling Considering Neural Network based Battery Degradation

Cunzhi Zhao and Xingpeng Li, “Microgrid Optimal Energy Scheduling Considering Neural Network based Battery Degradation”, IEEE Transactions on Power Systems, early access, Jan. 2023

work page 2023

[5] [5]

An Alternative Method for Solving Security-Constraint Unit Commitment with Neural Network Based Battery Degradation Model

Cunzhi Zhao and Xingpeng Li, “An Alternative Method for Solving Security-Constraint Unit Commitment with Neural Network Based Battery Degradation Model”, 54th North American Power Symposium, Salt Lake City, UT, USA, Oct. 2022

work page 2022

[6] [6]

Power load forecasting in the spring festival based on feedforward neural network model,

R. ZhiChao, Y. Qiang, W. Haiyan, C. Chao and L. Yuan, "Power load forecasting in the spring festival based on feedforward neural network model," 2017 3rd IEEE International Conference on Computer and Communications (ICCC), Chengdu, China, 2017, pp. 2855-2858, doi: 10.1109/CompComm.2017.8323053

work page arXiv 2017

[7] [7]

A Comparative Analysis of Deep Learning Models for Short-Term Load Forecasting,

I. Ul Haq, M. Khalid and U. Manzoor, "A Comparative Analysis of Deep Learning Models for Short-Term Load Forecasting," 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), Auckland, New Zealand, 2023, pp. 1-7, doi: 10.1109/CASE56687.2023.10260626

arXiv 2023

[8] [8]

Comparison of Deep Learning-Based Methods for Electrical Load Forecasting

S., Angel T. et al. “Comparison of Deep Learning-Based Methods for Electrical Load Forecasting.” 2022 Third International Conference on Intelligent Computing Instrumentation and Control Technologies (ICICICT) (2022): 1598-1602

work page 2022