REVIEW 4 major objections 5 minor 17 references
Dynamics of Resource Allocation in O-RANs: An In-depth Exploration of On-Policy and Off-Policy Deep Reinforcement Learning for Real-Time Applications
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A replication rerun finds two DRL models beat greedy in O-RAN resource allocation, with PPO balancing energy and latency best.
desk verdict A replication effort that undermines itself: the results section and the abstract say opposite things about which DRL algorithm converges faster, and no quantitative evidence is provided. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the O-RAN cross-slice resource orchestrator cast as a Markov decision process: at each slicing window the agent observes the demands of video cameras and latency-tolerant users plus the current vO-DU loads, chooses which vO-DU serves the incoming request, and receives a reward equal to the negative weighted sum of vO-DU power consumption $-\alpha \sum_m P_t^m$. The power model $P_t^m = P(0\%) + (P(100\%)-P(0\%))(2z_t^m-(z_t^m)^{1.4})$, taken from the cited literature, is the same objective used in the mixed-integer programming optimum and the greedy baseline. PPO carries the argument through a clipped surrogate objective that keeps policy updates stable, while ACER exploits a replay buffer to reuse past transitions; the two mechanisms are what produce the claimed stability-versus-sample-efficiency contrast.
What would settle it
Use the original code to retrain both agents with published hyperparameters and fixed random seeds, then check per-episode rewards against the reported curves: if either DRL agent fails to beat the greedy baseline, or if ACER converges no faster than PPO, the replication claim is false.
Extended reading notes
Core claim
The central claim is that the replication confirms the original study: a PPO-based on-policy orchestrator and an ACER-based off-policy orchestrator both reduce the power consumption of the vO-DU pool compared with a greedy heuristic, while respecting the latency constraint of real-time video surveillance. In head-to-head comparison, PPO approaches the optimal energy-latency trade-off more closely than ACER, and ACER reaches a stable policy faster in training. The paper also reports that PPO's training performance is stable across different dense-layer sizes, whereas ACER's is not, which matters when an operator must choose a model under hardware or time constraints.
Load-bearing premise
The conclusions stand only if the original study's code and experimental configuration, which this paper references rather than re-specifies, are correct and if a three-hour training run with unreported hyperparameters truly matches that setup.
Editorial extensions
If this is right
- O-RAN operators can realistically choose PPO when the goal is to minimize energy without exceeding the latency budget of real-time services.
- ACER becomes attractive when training time is scarce, since it converges faster even if it is more sensitive to network architecture.
- The comparison template of DRL agent versus MIP optimum versus greedy baseline gives a reusable yardstick for evaluating any future resource-allocation policy in O-RAN slicing.
- If the replication is truthful, the original result is robust to a different three-hour training run and, for PPO, to variations in dense-layer size.
Reading between the lines
- The paper's own evidence stops at aggregate training curves; a sharper test would publish the trained weights and per-run random seeds so another team can quantify how much of the gap between PPO and ACER is algorithmic rather than noise.
- The same MDP could be extended to mixed service classes with different latency budgets, but the paper does not test whether PPO's balance holds when the reward weights change.
- A practical implication the authors leave implicit is that the faster-converging ACER might be preferable for online retraining in a non-real-time RIC, where traffic drift forces periodic model updates.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports a replication study of Hammami and Nguyen's work on on-policy (PPO) and off-policy (ACER) deep reinforcement learning for resource allocation in sliced O-RAN systems. The authors re-use the original model and code (referenced only through a GitHub link), train both DRL agents for three hours, and compare their energy consumption and latency trade-off against a greedy baseline and a MIP-derived optimum. The abstract and conclusion claim that the results confirm the original findings: both DRL models outperform a greedy algorithm, PPO gives a favorable balance between energy consumption and user latency, and ACER converges faster. The evaluation section, however, states the opposite convergence result, reporting that PPO converged faster than ACER. The paper provides no quantitative metrics, error bars, seeds, or statistical tests to support any of these qualitative claims, and it does not numerically report the MIP or greedy baseline results to which the DRL models are compared.
Significance. A faithful, independent replication of a published O-RAN resource-allocation study would be valuable to the community, and the authors deserve credit for attempting to reproduce prior results on the Alibaba cluster-trace dataset and for applying two DRL paradigms in the same experimental framework. However, the manuscript as written provides no useful checkable evidence: the only explicit statement about convergence in the evaluation contradicts the paper's own abstract and conclusion; no convergence metric is defined; no numerical values, confidence intervals, or statistical tests are reported; and the study is not an independent implementation but a re-run of the original authors' model. The central replication claims are therefore unsupported and internally inconsistent, and the paper's potential significance cannot be assessed from the presented material.
major comments (4)
- [Section V(a), Abstract, Section VI] The paper contains a direct and load-bearing contradiction about which algorithm converges faster. Section V(a) states "we observed a faster convergence for the PPO model compared to ACER," while the abstract states "the off-policy model (ACER) shows a faster convergence" and Section VI concludes "ACER achieves faster convergence." These statements cannot both be true. The paper defines no convergence metric (episodes to threshold, wall-clock time, sample complexity), reports no quantitative convergence values, and provides no error bars or seed statistics for Figures 3-4. Since the replication's stated purpose is to validate the original observations, this unresolved contradiction makes the central finding uncheckable and unsupported.
- [Section V] No quantitative results are reported anywhere in the evaluation. The claims that both DRL models "showed superior performance compared to the greedy baseline," that "PPO showed the most efficient energy consumption, which resembles the optimal solution," and that PPO "showed better optimization in balancing latency and energy efficiency" are made without any numbers, such as mean power consumption, average latency, percentage improvement, or confidence intervals. Figures 3-8 are qualitative line plots with no error bands or multiple-seed statistics. The absence of quantitative evidence means the reader cannot independently verify the main comparative claims, and the paper's stated aim of "proving the efficacy" of the models is not met.
- [Section V, Ref [16]] The replication is not independent. The text states "we used the available model proposed in the original research" and cites only a GitHub repository (reference [16]), with no description of the original model, no list of modifications made by the authors, no hyperparameters for PPO or ACER beyond Table II (which omits e.g. clipping ratio, entropy coefficient, and replay buffer size), and only the statement that training lasted three hours. A replication that reuses the same implementation and experimental setup cannot by itself validate the original findings unless it provides an independent implementation or at least a detailed comparison of training configurations. As presented, the findings are generated by the same artifacts under test, so the claimed "confirmation" risks circularity rather than providing independent evidence.
- [Section V, Section III-B] The comparison against the MIP optimal solution and the greedy baseline is not verifiable. Section V mentions that the setup includes "the optimal solution derived from the MIP model which uses the Gurobi solver, and the same for the greedy baseline approach," but no table or figure reports the MIP and greedy numerical values, and the paper does not state over how many instances or under what conditions these baselines were computed. Without those data, statements such as PPO's energy consumption "resembles the optimal solution" are unsupported. The paper should provide at least one table with the mean and standard deviation of power consumption and latency for MIP, greedy, PPO, and ACER across multiple seeds and across the different network load levels shown in Figures 7-8.
minor comments (5)
- [Section IV, paragraph before 'ACER'] The paper misnames ACER in the text: it introduces "Advantage Actor-Critic (ACER)" but ACER stands for Actor-Critic with Experience Replay, as correctly stated in the abstract and in reference [14]. The naming should be corrected throughout.
- [Equation (3)] Equation (3) contains a typo in the subscript: "zt,m" should be "z_t^m" to match the notation used in the power consumption formula and elsewhere.
- [Section V] The paper states "we used the available model proposed in the original research and trained them for a duration of three hours" without specifying the hardware, the number of episodes or steps, or the number of independent runs. These details are necessary for any reproducibility claim.
- [References] Reference [16] is only a GitHub URL; the manuscript should cite the original paper by Hammami and Nguyen with full publication details, and should state explicitly what version of the code was used and whether any modifications were made.
- [Section VI] The conclusion states that the study "proves" the efficacy of the DRL models; given the qualitative nature of the results and the absence of statistical analysis, a claim of proof is too strong and should be tempered to "supports" or "is consistent with."
Circularity Check
The validation is self-referential: original model/code is both the object under test and the evidence source, and the convergence conclusion is contradicted inside the paper.
-
other
[Section V (Evaluation and Results) and Section VI (Conclusion)]
"In our replication study, we used the available model proposed in the original research and trained them for a duration of three hours. The results are similar to those of the original paper ... The findings align closely with those of the original study, proving the efficacy of the proposed DRL models in addressing resource allocation challenges in O-RAN systems."
The central conclusion that the replication confirms the original observations is not independently derived: the only new evidence is produced by running the original authors' model and setup from GitHub (ref [16]). No quantitative convergence metric, seeds, or numerical values are given; Figures 3-8 are described without data. Thus 'the results are similar' restates the behavior of the same artifact under test rather than providing an independent check. The phrase 'proving the efficacy' is therefore a self-referential validation: the model is used to validate the model.
full rationale
The mathematical content (MIP formulation, MDP definition, PPO/ACER descriptions) is standard and not circular in its equations. The circularity is in the validation strategy: the paper states it used the original model from the original research and then concludes that the findings align with the original study, proving the efficacy of the proposed DRL models. Because the same code and setup are both the object under test and the source of evidence, the replication cannot independently support the original claims. The paper also defines no convergence metric and does not report numeric results, and its only convergence sentence in Section V (PPO converged faster) contradicts the Abstract and Conclusion (ACER converges faster). This contradiction means the load-bearing claim about which algorithm converges faster cannot be checked from the presented evidence. The score is set at 6 rather than higher because the paper does contain independent components (Alibaba cluster trace, Gurobi MIP solver, greedy baseline) and because the issue is a self-referential replication claim combined with an internal contradiction, not a fully definitional equivalence of equations.
Assumptions & free parameters
free parameters (6)
- actor learning rate =
0.0003
- critic learning rate =
0.001
- gamma (discount factor) =
0.99
- alpha (reward weight) =
0.0001
- dense layer neuron counts
- P(0%) and P(100%) =
87W, 145W
assumptions (5)
- domain assumption Latency-tolerant user traffic follows an M/M/1 queue, and VSC packets follow a D/D/1 queue with mu > lambda to ensure negligible queueing delay.
- domain assumption Traffic remains stationary and resource allocation decisions unchanged within each slicing window.
- ad hoc to paper The reward function is the negative weighted sum of instantaneous power consumption (R = -alpha * sum P_t^m).
- domain assumption Power consumption follows the reference model P(t) = P(0%) + (P(100%) - P(0%)) * (2z - z^1.4) from [11].
- domain assumption The finite state space and action space are sufficient for the MDP to represent the allocation problem.
Cite this review
Pith. "Pith review of Dynamics of Resource Allocation in O-RANs: An In-depth Exploration of On-Policy and Off-Policy Deep Reinforcement Learning for Real-Time Applications." pith.science (2026). https://pith.science/paper/R6L4T6NB
@misc{pith2026241201839,
author = {Pith},
title = {Pith review of: Dynamics of Resource Allocation in O-RANs: An In-depth Exploration of On-Policy and Off-Policy Deep Reinforcement Learning for Real-Time Applications},
year = {2026},
howpublished = {\url{https://pith.science/paper/R6L4T6NB}},
note = {Machine review of arXiv:2412.01839}
}
read the original abstract
Deep Reinforcement Learning (DRL) is a powerful tool used for addressing complex challenges in mobile networks. This paper investigates the application of two DRL models, on-policy and off-policy, in the field of resource allocation for Open Radio Access Networks (O-RAN). The on-policy model is the Proximal Policy Optimization (PPO), and the off-policy model is the Sample Efficient Actor-Critic with Experience Replay (ACER), which focuses on resolving the challenges of resource allocation associated with a Quality of Service (QoS) application that has strict requirements. Motivated by the original work of Nessrine Hammami and Kim Khoa Nguyen, this study is a replication to validate and prove the findings. Both PPO and ACER are used within the same experimental setup to assess their performance in a scenario of latency-sensitive and latency-tolerant users and compare them. The aim is to verify the efficacy of on-policy and off-policy DRL models in the context of O-RAN resource allocation. Results from this replication contribute to the ongoing scientific research and offer insights into the reproducibility and generalizability of the original research. This analysis reaffirms that both on-policy and off-policy DRL models have better performance than greedy algorithms in O-RAN settings. In addition, it confirms the original observations that the on-policy model (PPO) gives a favorable balance between energy consumption and user latency, while the off-policy model (ACER) shows a faster convergence. These findings give good insights to optimize resource allocation strategies in O-RANs. Index Terms: 5G, O-RAN, resource allocation, ML, DRL, PPO, ACER.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[16]
Nessrine Hammami. ”On-policy vs. off-policy deep reinforcement learn- ing for resource allocation in open radio access network.” 2021. [online] Available: https://github.com/nessry/ON-OFF-DRL
work page 2021
-
[1]
Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansion
Bo Zhou, Hongsheng Zeng, Fan Wang, Yunxiang Li, and Hao Tian. ”Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansion.” arXiv preprint arXiv:1912.05328v1 [cs.LG] , 2019. https://arxiv.org/abs/1912.05328
work page Pith review arXiv 1912
-
[2]
Reinforcement learning with experience replay and adaptation of action dispersion
Paweł Wawrzy ´nski, Wojciech Masarczyk, and Mateusz Ostaszewski. ”Reinforcement learning with experience replay and adaptation of action dispersion.” arXiv:2208.00156, 2022
work page Pith review arXiv 2022
-
[3]
Zhiyuan Xu, Yanzhi Wang, Jian Tang, Jing Wang, and Mustafa Cenk Gursoy. ”A deep reinforcement learning based framework for power- efficient resource allocation in cloud RANs.” In 2017 IEEE International Conference, 2017
work page 2017
- [4]
-
[5]
Kunpeng Yang et al. ”Reinforcement learning-based mobile edge com- puting and transmission scheduling for video surveillance.” IEEE Trans- actions on Emerging Topics in Computing , pages 1–1, 2021
work page 2021
-
[6]
Z. Ming et al. ”Edge-Based Video Surveillance With Graph-Assisted Reinforcement Learning in Smart Construction.”IEEE Internet of Things Journal, 9(12):9249-9265, June 15, 2022. https://doi.org/10.1109/JIOT. 2021.3090513
arXiv 2022
-
[7]
Yujie Yao, Hao Zhou, and Melike Erol-Kantarci. ”Deep Reinforcement Learning-based Radio Resource Allocation and Beam Management under Location Uncertainty in 5G mmWave Networks.” School of Electrical Engineering and Computer Science, University of Ottawa,
Show all 17 references
-
[8]
”On-Policy vs
Matthew Hausknecht and Peter Stone. ”On-Policy vs. Off-Policy Up- dates for Deep Reinforcement Learning.” Published in Deep Rein- forcement Learning: Frontiers and Challenges, IJCAI 2016 Workshop , University of Texas at Austin. https://www.cs.utexas.edu/users/pstone/ Papers/b...
2016
-
[9]
S.-P. Yeh, S. Bhattacharya, R. Sharma, and H. Moustafa. ”Deep Learning for Intelligent and Automated Network Slicing in 5G Open RAN (ORAN) Deployment.” IEEE Open Journal of the Communications Society, 5:64-70, 2024. https://doi.org/10.1109/OJCOMS.2023.3337854
2024
-
[10]
Chia-Hsiang Lai, Li-Hsiang Shen, and Kai-Ten Feng. ”Intelligent Load Balancing and Resource Allocation in O-RAN: A Multi-Agent Multi- Armed Bandit Approach.” Department of Electronics and Electrical Engineering, National Yang Ming Chiao Tung University, Hsinchu, Taiwan, 2023. ...
2023 arXiv
-
[11]
Ning Liu, Zhe Li, Jielong Xu, Zhiyuan Xu, Sheng Lin, Qinru Qiu, Jian Tang, and Yanzhi Wang. ”A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning.” In 2017 IEEE 37th International Conference on Distributed Computing Syste...
2017
-
[12]
”Deep reinforcement learning for resource management in network slicing.” IEEE Access , 6:74429–74441, 2018
Rongpeng Li, Zhifeng Zhao, Qi Sun, I Chih-Lin, Chenyang Yang, Xianfu Chen, Minjian Zhao, and Honggang Zhang. ”Deep reinforcement learning for resource management in network slicing.” IEEE Access , 6:74429–74441, 2018
2018
-
[13]
”Proximal policy optimization algorithms.” ArXiv, abs/1707.06347, 2017
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. ”Proximal policy optimization algorithms.” ArXiv, abs/1707.06347, 2017
2017 arXiv
-
[14]
”Sample efficient actor-critic with experience replay.” ArXiv, abs/1611.01224, 2017
Ziyu Wang, Victor Bapst, Nicolas Heess, V olodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. ”Sample efficient actor-critic with experience replay.” ArXiv, abs/1611.01224, 2017
2017 arXiv
-
[15]
[online] Available: https: //github.com/alibaba/clusterdata/tree/master/cluster-trace-v2018/
Alibaba production cluster data v2018, 2018. [online] Available: https: //github.com/alibaba/clusterdata/tree/master/cluster-trace-v2018/
2018
-
[2024]
https://arxiv.org/pdf/2204.10984.pdf
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.