REVIEW 3 major objections 7 minor 31 references
A View on Deep Reinforcement Learning in System Optimization
T0 review · 3 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Deep reinforcement learning for system optimization often lacks a convincing case: many prior works skip state-of-the-art comparisons or do worse than simple baselines, and the authors propose a standard evaluation checklist to fix this.
desk verdict A useful evaluation checklist for RL-in-systems papers, but the survey's empirical table is weaker than the prose indicates and should not be cited as proof that deep RL performs worse. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the two-table survey: Table 1 codifies how each reviewed work formulates the problem as states, actions, rewards, algorithm, and neural-network model, while Table 2 records environment step time, number of samples per iteration, training iterations, total steps, whether the method improves the state of the art, and whether it compares against bandit or random-search baselines. The Section 5 checklist—eight questions covering MDP suitability, whether the problem is truly RL, delayed rewards, what is being learned, random-search and bandit comparisons, expert actions, reproducibility and generalization, and state-of-the-art performance—turns this tabular evidence into a prescription for evaluating future deep RL systems work.
What would settle it
A reader could falsify the central claim by conducting a systematic review with explicit inclusion criteria and reimplementing a sample of deep RL system-optimization systems on shared benchmarks; if most achieved large, reproducible gains over random search and bandit baselines and outperformed existing heuristics, the paper's conclusion that deep RL often does not beat baselines would be overturned.
Extended reading notes
Core claim
The paper's central assessment is that the recent wave of deep RL in systems is, in many cases, an answer in search of a problem: Table 2 shows that for numerous reviewed works no comparison to the state of the art is available (marked "no comparison" or N/A), and in several cases deep RL performs only similarly to or worse than existing solutions. The authors further claim that many system optimization problems that appear to invite RL are actually one-step decision problems where bandits or greedy algorithms are sufficient or even superior, and that random search can match deep RL when the state features carry no learnable pattern. As a remedy, the paper proposes a set of evaluation questions in Section 5 that future work should answer, and it illustrates the checklist on a representative resource-allocation system to show how the challenges of slow real-system interaction, instability, reproducibility, and generalization manifest in practice.
Load-bearing premise
The verdict that deep RL often fails to beat baselines depends on the papers compiled in the review's tables being representative and their reported numbers accurate; the authors note that many works did not supply enough data to reproduce results and did not independently rerun the reported experiments.
Editorial extensions
If this is right
- Future deep RL papers in system optimization will face an expectation to include random-search and bandit baselines, not just domain-specific heuristics.
- The checklist gives reviewers a concrete language for judging submissions: an MDP framing is necessary but not sufficient, and delayed rewards must be shown to actually matter.
- Algorithm choice becomes more principled: model-based or temporal-difference methods are preferred when samples are costly, policy-gradient methods when simulators are fast, and bandits when the decision problem is one-step.
- If adopted, the checklist will likely shrink the set of problems for which deep RL is considered a justified choice, concentrating research effort on genuinely sequential tasks.
- The emphasis on reproducibility implies that papers without released code, workloads, or hyperparameters should be treated as incomplete evidence for a claimed RL gain.
Reading between the lines
- A consequence the authors leave implicit: the checklist could be applied retroactively as a scoring rubric, turning the qualitative gaps in Table 2 into a quantitative map of where deep RL has actually earned its keep.
- A testable extension: for each reviewed problem, replace the deep RL agent with a contextual bandit operating on the same state features and compare cumulative reward; problems where the bandit matches RL would be better modeled as one-step decisions.
- The paper's sample-efficiency ranking suggests that progress in this area will come more from model-based RL and calibrated simulators than from new policy-gradient variants.
- If the checklist becomes a community norm, it could drive a convergence toward shared benchmarks and reproducible baselines, which would make each new claimed improvement far easier to verify.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper is a critical survey and position statement on the use of deep reinforcement learning (deep RL) for system optimization. It reviews roughly twenty prior works spanning congestion control, packet classification, resource allocation, query optimization, compiler phase ordering, and device placement. The authors propose a checklist of evaluation questions (Section 5) intended to help systems researchers decide whether deep RL is appropriate for a given problem and how to report results so that they are reproducible and comparable. They argue that in many published works, comparisons against the state of the art are missing or the reported gains are modest, and that random search, bandit, or greedy baselines often provide a stronger or equally strong reference point. The paper also discusses challenges (slow environment interactions, instability, poor reproducibility, ill-defined reward/state/action spaces, lack of generalization, and missing standardized benchmarks) and illustrates the proposed checklist on the DeepRM resource-management case study.
Significance. If the paper's assessment is accepted, it provides a valuable corrective to the rapidly growing but often weakly evaluated literature on deep RL for systems. The proposed checklist is actionable and could shift the default evaluation protocol for this subfield: requiring comparisons against random search, bandit baselines, and the existing state of the art, together with reproducible artifacts. The paper is honest about the limitations of the underlying literature, explicitly noting that many prior works do not provide enough data to be reproduced. The DeepRM case study concretely demonstrates how the checklist can be applied. The survey does not introduce new algorithms or results, but its contribution as a critical synthesis and evaluation guide is significant for the systems-for-ML and ML-for-systems community.
major comments (3)
- [Section 5, last bullet; Table 2] The claim that 'deep RL performs worse' in many of the works listed in Table 2 is not supported by the table as printed. Table 2 contains no row in which the reported improvement is negative or marked 'worse'; the closest entries are 'similar' (Jay et al. 2019, Zhong et al. 2017, Huang et al. 2019) and numerous 'no comparison' entries. A statement that a comparison is unavailable is not evidence of worse performance. Please either cite specific works with documented worse-than-baseline results (e.g., the random-search observation for Huang et al. discussed in Section 5) or revise the wording to say that comparisons are often absent and that some works report only similar performance.
- [Section 4, Tables 1 and 2] The survey does not state inclusion or exclusion criteria for the works in Tables 1 and 2, and the boundary of what counts as 'deep RL' is applied inconsistently. Section 2.1 discusses non-deep approximation methods (tables, linear approximators), yet Table 1 includes entries such as Tesauro et al. (2006), Rao et al. (2009), Xu et al. (2012), Coons et al. (2008), and Kulkarni et al. (2012) whose algorithms appear to be classical RL or evolutionary search rather than deep neural network training in the sense defined in the introduction. Because the central negative assessment in Section 5 is supported by aggregate statements about Table 2, the absence of a clear deep/classical boundary weakens the evidential basis. Please add explicit inclusion criteria, justify the classification of each row (or restrict the tables to works that satisfy the paper's own deep-RL definition), and clarify how the 'Improves State of the Art' and 'Compares Against Bandit/Random Search' columns were populated.
- [Section 4.1] The paper acknowledges that 'many works did not provide sufficient data to reproduce the results,' but it does not describe how the quantitative entries in Table 2 (e.g., percentages of improvement, step counts) were obtained from the cited papers. Were these numbers taken at face value from the original papers, re-computed from reported data, or estimated? Without this methodology note, the reader cannot judge the reliability of the summary statistics that the paper's conclusions rely on. Please add a short paragraph describing the data-collection and verification process for Table 2.
minor comments (7)
- [Table 1 caption] The caption says 'Problem formulation in the deep RL setting,' but several listed works rely on classical RL or evolutionary methods; consider rewording the caption to 'Problem formulation in the RL setting considered in this review' or adjusting the table contents to match the deep-RL scope.
- [Section 3.3] The sentence 'Continuous policies can handle both continuous and episodic tasks, while episodic policies cannot' is confusing: 'continuous' here appears to mean continuing tasks, not continuous action spaces. Please rephrase to avoid ambiguity.
- [Section 2, Eq. (1)] The notation in Equation (1) is informal: the expectation is written over trajectories but the policy parameter is not explicitly conditioned on theta in the definition. Clarify that the expectation is with respect to the trajectory distribution induced by the policy.
- [Section 4.1] There is a typo: 'the the cost of each environment step' should read 'the cost of each environment step.'
- [Section 8] There is a repeated word: 'the sum of of job slowdowns' should read 'the sum of job slowdowns.'
- [Section 5] The checklist items would be easier to reference if they were numbered or labeled with short headings; consider adding marginal labels such as 'MDP suitability,' 'Delay structure,' and 'Baseline comparison.'
- [Table 2] The formatting of Table 2 is difficult to parse in the PDF: columns run together and checkmarks are not aligned. Please ensure the camera-ready version has clearly separated columns and a legend for the checkmark symbols.
Circularity Check
No circularity: the paper is a survey whose checklist is normative, not a derivation from fitted inputs.
full rationale
The paper does not claim to derive any predicted quantity from fitted parameters. Section 5's evaluation checklist is presented as a normative proposal ('we propose a set of questions that can help system optimization researchers determine whether deep RL could be an effective tool'), not as a mathematical consequence of a model. Section 4.1 reports qualitative observations about the reviewed literature and explicitly notes missing data ('many works did not provide sufficient data to reproduce the results'), which is an evidentiary limitation rather than a circular step. The few self-citations (e.g., Huang et al. 2019 as an example where random search gave a 10% improvement) are used as anecdotal evidence inside a survey, not as an unverified uniqueness theorem or as the premise that forces the conclusion. No equation in the paper transforms an input into an output by construction, and no fitted value is renamed as a prediction. Any concern about Table 1/Table 2 mixing non-deep RL works into the 'deep RL' evidence base concerns sampling validity or correctness of the survey's empirical summary, not circularity, because the conclusion does not define or fit the table entries; the table is external evidence being summarized. The scoring rubric therefore supports a non-finding.
Assumptions & free parameters
assumptions (2)
- domain assumption System optimization problems under review can be adequately modeled as Markov Decision Processes with defined states, actions, and rewards.
- domain assumption The reported results and numbers in the reviewed papers, especially those compiled in Table 2, are accurate as cited.
Cite this review
Pith. "Pith review of A View on Deep Reinforcement Learning in System Optimization." pith.science (2026). https://pith.science/paper/CU3TPNES
@misc{pith2026190801275,
author = {Pith},
title = {Pith review of: A View on Deep Reinforcement Learning in System Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/CU3TPNES}},
note = {Machine review of arXiv:1908.01275}
}
read the original abstract
Many real-world systems problems require reasoning about the long term consequences of actions taken to configure and manage the system. These problems with delayed and often sequentially aggregated reward, are often inherently reinforcement learning problems and present the opportunity to leverage the recent substantial advances in deep reinforcement learning. However, in some cases, it is not clear why deep reinforcement learning is a good fit for the problem. Sometimes, it does not perform better than the state-of-the-art solutions. And in other cases, random search or greedy algorithms could outperform deep reinforcement learning. In this paper, we review, discuss, and evaluate the recent trends of using deep reinforcement learning in system optimization. We propose a set of essential metrics to guide future works in evaluating the efficacy of using deep reinforcement learning in system optimization. Our evaluation includes challenges, the types of problems, their formulation in the deep reinforcement learning setting, embedding, the model used, efficiency, and robustness. We conclude with a discussion on open challenges and potential directions for pushing further the integration of reinforcement learning in system optimization.
Figures
Reference graph
Works this paper leans on
-
[1]
B., Gupta, S., Mao, H., and Alizadeh, M
Addanki, R., Venkatakrishnan, S. B., Gupta, S., Mao, H., and Alizadeh, M. Placeto: Learning generalizable device placement algorithms for distributed machine learning. arXiv preprint arXiv:1906.08879,
arXiv 1906
-
[6]
Speech recog- nition with deep recurrent neural networks
Graves, A., Mohamed, A.-r., and Hinton, G. Speech recog- nition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645–6649. IEEE,
work page 2013
-
[7]
Guu, K., Pasupat, P., Liu, E. Z., and Liang, P. From language to programs: Bridging reinforcement learn- ing and maximum marginal likelihood. arXiv preprint arXiv:1704.07926,
-
[8]
Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. arXiv preprint arXiv:1801.01290,
-
[9]
Autophase: Compiler phase-ordering for hls with deep reinforcement learn- ing
Huang, Q., Haj-Ali, A., Moses, W., Xiang, J., Stoica, I., Asanovic, K., and Wawrzynek, J. Autophase: Compiler phase-ordering for hls with deep reinforcement learn- ing. In 2019 IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pp. 308–308. IEEE,
work page 2019
-
[12]
Learning- based and data-driven tcp design for memory-constrained iot
Li, W., Zhou, F., Meleis, W., and Chowdhury, K. Learning- based and data-driven tcp design for memory-constrained iot. In 2016 International Conference on Distributed Computing in Sensor Systems (DCOSS) , pp. 199–205. IEEE,
work page 2016
-
[14]
Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Gon- zalez, J., Goldberg, K., and Stoica, I. Ray rllib: A com- posable and scalable reinforcement learning library.arXiv preprint arXiv:1712.09381,
-
[15]
Liang, E., Zhu, H., Jin, X., and Stoica, I. Neural packet classification. arXiv preprint arXiv:1902.10319,
work page Pith review arXiv 1902
Show all 31 references
-
[18]
Neo: A learned query optimizer
Marcus, R., Negi, P., Mao, H., Zhang, C., Alizadeh, M., Kraska, T., Papaemmanouil, O., and Tatbul, N. Neo: A learned query optimizer. arXiv preprint arXiv:1904.03711,
1904 arXiv
-
[19]
Playing atari with deep reinforcement learning
Mnih, V ., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602,
-
[20]
P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K
Mnih, V ., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asyn- chronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928– 1937,
1928
-
[21]
Mostafavi, S., Ahmadi, F., and Sarram, M. A. Reinforcement-learning-based foresighted task schedul- ing in cloud computing.arXiv preprint arXiv:1810.04718,
-
[22]
Ortiz, J., Balazinska, M., Gehrke, J., and Keerthi, S. S. Learning state representations for query optimization with deep reinforcement learning. arXiv preprint arXiv:1803.08604,
-
[23]
Regal: Transfer learning for fast optimization of computation graphs
Paliwal, A., Gimeno, F., Nair, V ., Li, Y ., Lubin, M., Kohli, P., and Vinyals, O. Regal: Transfer learning for fast optimization of computation graphs. arXiv preprint arXiv:1905.02494,
1905 arXiv
-
[24]
Semantic locality and context-based prefetching using reinforce- ment learning
Peled, L., Mannor, S., Weiser, U., and Etsion, Y . Semantic locality and context-based prefetching using reinforce- ment learning. In 2015 ACM/IEEE 42nd Annual Interna- tional Symposium on Computer Architecture (ISCA), pp. 285–297. IEEE,
2015
-
[27]
P., Obraczka, K., Burleigh, S., and Hirata, C
Silva, A. P., Obraczka, K., Burleigh, S., and Hirata, C. M. Smart congestion control for delay-and disruption toler- ant networks. In 2016 13th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), pp. 1–9. IEEE,
2016
-
[28]
Machine learning in compiler optimization
Wang, Z., and OBoyle, M. Machine learning in compiler optimization. Proceedings of the IEEE, 106(11):1879– 1901,
1901
-
[30]
Seq2sql: Generating structured queries from natural language using reinforce- ment learning
Zhong, V ., Xiong, C., and Socher, R. Seq2sql: Generating structured queries from natural language using reinforce- ment learning. arXiv preprint arXiv:1709.00103,
-
[31]
A reinforcement learning approach to automatic error recovery
Zhu, Q., and Yuan, C. A reinforcement learning approach to automatic error recovery. In 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07), pp. 729–738. IEEE, 2007
2007
-
[1999]
Proximal policy optimization algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
-
[2000]
Energy-efficient virtual machines consolidation in cloud data centers us- ing reinforcement learning
Farahnakian, F., Liljeberg, P., and Plosila, J. Energy-efficient virtual machines consolidation in cloud data centers us- ing reinforcement learning. In 2014 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, pp. 500–507. IEEE,
2014
-
[2002]
Learning to optimize join queries with deep reinforcement learning
Krishnan, S., Yang, Z., Goldberg, K., Hellerstein, J., and Stoica, I. Learning to optimize join queries with deep reinforcement learning. arXiv preprint arXiv:1808.03196,
-
[2008]
M., Pahl, C., Metzger, A., and Estrada, G
Jamshidi, P., Sharifloo, A. M., Pahl, C., Metzger, A., and Estrada, G. Self-learning cloud controllers: Fuzzy q- learning for knowledge evolution. In 2015 International Conference on Cloud and Autonomic Computing, pp. 208–
2015
-
[2011]
Iroko: A framework to prototype reinforcement learning for data center traffic control
Ruffy, F., Przystupa, M., and Beschastnikh, I. Iroko: A framework to prototype reinforcement learning for data center traffic control. arXiv preprint arXiv:1812.09975,
-
[2012]
Xu, Z., Wang, Y ., Tang, J., Wang, J., and Gursoy, M. C. A deep reinforcement learning based framework for power- efficient resource allocation in cloud rans. In 2017 IEEE International Conference on Communications (ICC), pp. 1–6. IEEE,
2017
-
[2013]
A hierarchical framework of cloud resource allo- cation and power management using deep reinforcement learning
Liu, N., Li, Z., Xu, J., Xu, Z., Lin, S., Qiu, Q., Tang, J., and Wang, Y . A hierarchical framework of cloud resource allo- cation and power management using deep reinforcement learning. In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pp. 3...
2017
-
[2014]
Horizon: Facebook’s open source applied reinforcement learning platform
A View on Deep Reinforcement Learning in System Optimization Gauci, J., Conti, E., Liang, Y ., Virochsiri, K., He, Y ., Kaden, Z., Narayanan, V ., and Ye, X. Horizon: Facebook’s open source applied reinforcement learning platform. arXiv preprint arXiv:1811.00260,
-
[2016]
D., and Lao, N
A View on Deep Reinforcement Learning in System Optimization Liang, C., Berant, J., Le, Q., Forbus, K. D., and Lao, N. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. arXiv preprint arXiv:1611.00020,
-
[2017]
Castro, P
URLhttps: //doi.org/10.5281/zenodo.1134899. Castro, P. S., Moitra, S., Gelada, C., Kumar, S., and Belle- mare, M. G. Dopamine: A Research Framework for Deep Reinforcement Learning
-
[2018]
URL http: //arxiv.org/abs/1812.06110. Choi, S. P., and Yeung, D.-Y . Predictive q-routing: A memory-based reinforcement learning approach to adap- tive traffic control. In Advances in Neural Information Processing Systems, pp. 945–951,
-
[2019]
P., Hunt, J
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y ., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971,
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.