{"id":"a7438896-a10c-42c1-8334-407bb2d097af","arxiv_id":"1908.01275","paper_version":3,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of deep RL in system optimization finds that many published solutions lack strong baselines, reproducibility, and comparisons to random search, and proposes evaluation questions to address this.","lead":"This paper is a survey of deep reinforcement learning applied to computer system optimization problems such as cloud scheduling, packet classification, and SQL query optimization. It documents frequent failures of deep RL to beat simple baselines and proposes a checklist of evaluation metrics for future work.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1/2 mix non-deep RL works into the 'deep RL' evidence base, so the Section 5 burden-shift claim is partly built on classical methods rather than deep RL.","rationale":"The reader correctly identified Table 2's accuracy and representativeness as the weak point, but the more specific and damaging issue is internal: the paper's own evidence base is contaminated with non-deep RL works. This matters because the central claim, as interpreted by the reader, is that the burden of proof should shift for deep RL applications. If many of the tabulated failures or missing comparisons come from classical RL methods, the empirical foundation for that shift is not established for deep RL. The normative checklist in Section 5 remains reasonable and can stand, but the survey's factual assertions need to be re-scoped. A conditional acceptance—requiring the authors to state inclusion criteria, filter Table 1/2 to genuine deep RL, and correct the unsupported 'performs worse' phrasing—would preserve the paper's contribution while removing the overreach. This is not a rejection because the proposed metrics and the illustrative DeepRM analysis are still useful independent of the exact survey counts.","tokens_in":18484,"tokens_out":8803,"duration_ms":91999,"concrete_test":"For each row in Table 1, read the cited paper's methods section and classify the function approximator as: (a) multi-layer neural network trained by gradient-based deep RL (deep), or (b) tabular, linear, single-layer, or evolutionary method (non-deep). Rebuild Table 2 using only classes (a) and recompute: (i) how many deep RL works provide a state-of-the-art comparison, (ii) how many compare against random search or a bandit, and (iii) how many report inferior, similar, or superior results. If, after exclusion, the proportion of deep RL works lacking baseline comparisons is still a clear majority, the burden-shift claim survives; if the proportion drops materially or the 'worse' cases disappear, Sections 4 and 5 must be narrowed to avoid attributing non-deep RL results to deep RL.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central negative assessment (Section 5: 'In many prior works listed in Table 2 a comparison against the state of the art is not available or deep RL performs worse') and the resulting burden-shift recommendation depend on Table 1 and Table 2 being an accurate sample of deep RL applications. But Table 1 is titled 'Problem formulation in the deep RL setting' and includes entries that Section 2.1 itself classifies as non-deep: Tesauro et al. (2006), Rao et al. (2009), Xu et al. (2012), Coons et al. (2008), and Kulkarni et al. (2012) use tabular/linear approximations or evolutionary methods, not multi-layer neural networks trained with deep RL. These rows feed directly into Table 2's counts of missing comparisons and 'similar' results, so the statement that deep RL specifically often fails to beat baselines conflates deep RL with classical RL/function approximation. The paper offers no inclusion criteria or per-row justification for what qualifies as 'deep,' and Section 4.1 only notes that 'many works did not provide sufficient data to reproduce the results' without addressing the sampling boundary. Additionally, Table 2 contains no row marked 'worse,' so the 'deep RL performs worse' clause in Section 5 is not actually evidenced by the table as printed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a critical survey and position statement on the use of deep reinforcement learning (deep RL) for system optimization. It reviews roughly twenty prior works spanning congestion control, packet classification, resource allocation, query optimization, compiler phase ordering, and device placement. The authors propose a checklist of evaluation questions (Section 5) intended to help systems researchers decide whether deep RL is appropriate for a given problem and how to report results so that they are reproducible and comparable. They argue that in many published works, comparisons against the state of the art are missing or the reported gains are modest, and that random search, bandit, or greedy baselines often provide a stronger or equally strong reference point. The paper also discusses challenges (slow environment interactions, instability, poor reproducibility, ill-defined reward/state/action spaces, lack of generalization, and missing standardized benchmarks) and illustrates the proposed checklist on the DeepRM resource-management case study.","tokens_in":18716,"tokens_out":7232,"duration_ms":65272,"significance":"If the paper's assessment is accepted, it provides a valuable corrective to the rapidly growing but often weakly evaluated literature on deep RL for systems. The proposed checklist is actionable and could shift the default evaluation protocol for this subfield: requiring comparisons against random search, bandit baselines, and the existing state of the art, together with reproducible artifacts. The paper is honest about the limitations of the underlying literature, explicitly noting that many prior works do not provide enough data to be reproduced. The DeepRM case study concretely demonstrates how the checklist can be applied. The survey does not introduce new algorithms or results, but its contribution as a critical synthesis and evaluation guide is significant for the systems-for-ML and ML-for-systems community.","major_comments":[{"comment":"The claim that 'deep RL performs worse' in many of the works listed in Table 2 is not supported by the table as printed. Table 2 contains no row in which the reported improvement is negative or marked 'worse'; the closest entries are 'similar' (Jay et al. 2019, Zhong et al. 2017, Huang et al. 2019) and numerous 'no comparison' entries. A statement that a comparison is unavailable is not evidence of worse performance. Please either cite specific works with documented worse-than-baseline results (e.g., the random-search observation for Huang et al. discussed in Section 5) or revise the wording to say that comparisons are often absent and that some works report only similar performance.","section":"Section 5, last bullet; Table 2"},{"comment":"The survey does not state inclusion or exclusion criteria for the works in Tables 1 and 2, and the boundary of what counts as 'deep RL' is applied inconsistently. Section 2.1 discusses non-deep approximation methods (tables, linear approximators), yet Table 1 includes entries such as Tesauro et al. (2006), Rao et al. (2009), Xu et al. (2012), Coons et al. (2008), and Kulkarni et al. (2012) whose algorithms appear to be classical RL or evolutionary search rather than deep neural network training in the sense defined in the introduction. Because the central negative assessment in Section 5 is supported by aggregate statements about Table 2, the absence of a clear deep/classical boundary weakens the evidential basis. Please add explicit inclusion criteria, justify the classification of each row (or restrict the tables to works that satisfy the paper's own deep-RL definition), and clarify how the 'Improves State of the Art' and 'Compares Against Bandit/Random Search' columns were populated.","section":"Section 4, Tables 1 and 2"},{"comment":"The paper acknowledges that 'many works did not provide sufficient data to reproduce the results,' but it does not describe how the quantitative entries in Table 2 (e.g., percentages of improvement, step counts) were obtained from the cited papers. Were these numbers taken at face value from the original papers, re-computed from reported data, or estimated? Without this methodology note, the reader cannot judge the reliability of the summary statistics that the paper's conclusions rely on. Please add a short paragraph describing the data-collection and verification process for Table 2.","section":"Section 4.1"}],"minor_comments":[{"comment":"The caption says 'Problem formulation in the deep RL setting,' but several listed works rely on classical RL or evolutionary methods; consider rewording the caption to 'Problem formulation in the RL setting considered in this review' or adjusting the table contents to match the deep-RL scope.","section":"Table 1 caption"},{"comment":"The sentence 'Continuous policies can handle both continuous and episodic tasks, while episodic policies cannot' is confusing: 'continuous' here appears to mean continuing tasks, not continuous action spaces. Please rephrase to avoid ambiguity.","section":"Section 3.3"},{"comment":"The notation in Equation (1) is informal: the expectation is written over trajectories but the policy parameter is not explicitly conditioned on theta in the definition. Clarify that the expectation is with respect to the trajectory distribution induced by the policy.","section":"Section 2, Eq. (1)"},{"comment":"There is a typo: 'the the cost of each environment step' should read 'the cost of each environment step.'","section":"Section 4.1"},{"comment":"There is a repeated word: 'the sum of of job slowdowns' should read 'the sum of job slowdowns.'","section":"Section 8"},{"comment":"The checklist items would be easier to reference if they were numbered or labeled with short headings; consider adding marginal labels such as 'MDP suitability,' 'Delay structure,' and 'Baseline comparison.'","section":"Section 5"},{"comment":"The formatting of Table 2 is difficult to parse in the PDF: columns run together and checkmarks are not aligned. Please ensure the camera-ready version has clearly separated columns and a legend for the checkmark symbols.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The skeptical note's specific attribution that Section 2.1 itself classifies Tesauro et al., Rao et al., Xu et al., Coons et al., and Kulkarni et al. as non-deep is not exactly supported by the text: those references appear in Table 1, not in the Section 2.1 list. However, the broader point about an unclear deep/classical boundary is valid and is reflected in my second major comment. The paper's core contribution, the evaluation checklist, is sound and useful; the revisions I request concern the evidential basis for the paper's negative assessment, which is load-bearing for the main message. With a clarified scope and either strengthened evidence or carefully softened claims, the paper would be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful position piece with a practical evaluation checklist, but its central empirical claim is weaker than the prose suggests. The Table 2 evidence base mixes non-deep RL works and never actually shows a \"worse\" outcome, so the sentence \"deep RL performs worse\" overstates what the table demonstrates.\n\nThe genuinely new thing here is the checklist in Section 5: whether the problem is an MDP, whether rewards are delayed, whether you compare against random search and bandits, whether expert actions exist, whether results are reproducible, and whether you improve SOTA. That checklist is sensible, actionable, and likely to improve applied RL papers if adopted. I also credit the survey for collecting training sample counts and environment step times—information that is almost never reported in systems papers but is essential for judging whether RL is practical. The DeepRM walk-through is a nice concrete illustration of what the checklist looks like in practice.\n\nThe soft spots are real but moderate. As you noted, Table 1 is titled \"deep RL setting\" yet includes works that Section 2.1 itself categorizes as using tables, linear approximations, or evolutionary methods (Coons 2008, Kulkarni 2012, possibly Tesauro 2006, Rao 2009, Xu 2012). That means the \"in many prior works listed in Table 2\" claim conflates deep RL with classical RL/function approximation. The paper doesn't give inclusion criteria, so we can't tell if the sample is representative. Second, Table 2 has no row where the result is marked \"worse\"—it mostly shows \"similar\" or \"no comparison\". So the strong wording \"deep RL performs worse\" is not actually backed by the table as printed. These flaws don't destroy the paper's value: the checklist is normative and stands on its own, and the broader point that RL often fails to beat simple baselines is supported by the cited literature (e.g., Huang et al.'s random search result). But the authors should tighten the claim and either reclassify or justify the non-deep rows.\n\nWho is this for? Anyone working on RL for systems, and especially authors of applied RL papers who need a cheap self-check before submission. It's also useful for reviewers as a quick screen. I'd bring it to a reading group to debate the checklist and the evidence table. It deserves serious peer review: the topic is important, the authors are credible, and the proposal has real practical value. I'd accept it into the review process, with a request to fix the evidence table and align the prose with what the table actually shows.","headline":"A useful evaluation checklist for RL-in-systems papers, but the survey's empirical table is weaker than the prose indicates and should not be cited as proof that deep RL performs worse.","tokens_in":19259,"tokens_out":4224,"would_cite":true,"duration_ms":38966,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep reinforcement learning for system optimization often lacks a convincing case: many prior works skip state-of-the-art comparisons or do worse than simple baselines, and the authors propose a standard evaluation checklist to fix this.","keywords":["deep reinforcement learning","system optimization","evaluation metrics","baseline comparison","random search","reproducibility","MDP formulation","resource allocation"],"falsifier":"A reader could falsify the central claim by conducting a systematic review with explicit inclusion criteria and reimplementing a sample of deep RL system-optimization systems on shared benchmarks; if most achieved large, reproducible gains over random search and bandit baselines and outperformed existing heuristics, the paper's conclusion that deep RL often does not beat baselines would be overturned.","tokens_in":18297,"feed_emoji":"⚙️","tokens_out":5547,"duration_ms":58213,"temperature":0.7,"pith_summary":"This review argues that much recent work applying deep reinforcement learning to system optimization has not shown that deep RL is the right tool for the job. Across the surveyed papers, many lack any comparison against the state of the art, and where comparisons exist deep RL often ties with or loses to random search, greedy, or bandit methods. The authors propose a checklist of evaluation questions—can the problem be modeled as an MDP, are rewards delayed, does the method beat random search and a bandit, does it beat the state of the art, is it reproducible—and argue these should become the default standard for future work. The practical upshot is a shifted burden of proof: a deep RL solution should not be presented as a system optimization win unless it is measured against simple baselines and released in a reproducible way.","feed_headline":"Deep RL often loses to simple baselines in system optimization","feed_subtitle":"A survey of prior work finds many papers skip state-of-the-art comparisons; the fix is a standard evaluation checklist.","key_machinery":"The load-bearing object is the two-table survey: Table 1 codifies how each reviewed work formulates the problem as states, actions, rewards, algorithm, and neural-network model, while Table 2 records environment step time, number of samples per iteration, training iterations, total steps, whether the method improves the state of the art, and whether it compares against bandit or random-search baselines. The Section 5 checklist—eight questions covering MDP suitability, whether the problem is truly RL, delayed rewards, what is being learned, random-search and bandit comparisons, expert actions, reproducibility and generalization, and state-of-the-art performance—turns this tabular evidence into a prescription for evaluating future deep RL systems work.","core_discovery":"The paper's central assessment is that the recent wave of deep RL in systems is, in many cases, an answer in search of a problem: Table 2 shows that for numerous reviewed works no comparison to the state of the art is available (marked \"no comparison\" or N/A), and in several cases deep RL performs only similarly to or worse than existing solutions. The authors further claim that many system optimization problems that appear to invite RL are actually one-step decision problems where bandits or greedy algorithms are sufficient or even superior, and that random search can match deep RL when the state features carry no learnable pattern. As a remedy, the paper proposes a set of evaluation questions in Section 5 that future work should answer, and it illustrates the checklist on a representative resource-allocation system to show how the challenges of slow real-system interaction, instability, reproducibility, and generalization manifest in practice.","pith_inferences":["A consequence the authors leave implicit: the checklist could be applied retroactively as a scoring rubric, turning the qualitative gaps in Table 2 into a quantitative map of where deep RL has actually earned its keep.","A testable extension: for each reviewed problem, replace the deep RL agent with a contextual bandit operating on the same state features and compare cumulative reward; problems where the bandit matches RL would be better modeled as one-step decisions.","The paper's sample-efficiency ranking suggests that progress in this area will come more from model-based RL and calibrated simulators than from new policy-gradient variants.","If the checklist becomes a community norm, it could drive a convergence toward shared benchmarks and reproducible baselines, which would make each new claimed improvement far easier to verify."],"forward_implications":["Future deep RL papers in system optimization will face an expectation to include random-search and bandit baselines, not just domain-specific heuristics.","The checklist gives reviewers a concrete language for judging submissions: an MDP framing is necessary but not sufficient, and delayed rewards must be shown to actually matter.","Algorithm choice becomes more principled: model-based or temporal-difference methods are preferred when samples are costly, policy-gradient methods when simulators are fast, and bandits when the decision problem is one-step.","If adopted, the checklist will likely shrink the set of problems for which deep RL is considered a justified choice, concentrating research effort on genuinely sequential tasks.","The emphasis on reproducibility implies that papers without released code, workloads, or hyperparameters should be treated as incomplete evidence for a claimed RL gain."],"supporting_citations":[{"why":"Serves as the illustrative DeepRM case through which the paper applies its full evaluation checklist and shows the typical mix of simulated training, missing real-system generalization, and claims of state-of-the-art improvement.","marker":"Mao et al. (2016)"},{"why":"Provides the concrete evidence that random search alone achieved a 10% improvement over baseline, supporting the claim that simple search can match or beat deep RL.","marker":"Huang et al. (2019)"},{"why":"Cited as showing that changing only the random seed produces different results, grounding the paper's reproducibility and chance-results concerns.","marker":"Haarnoja et al. (2018)"},{"why":"Supplies the finite-time guarantees for multi-armed bandits that motivate comparing deep RL against bandit baselines in system optimization.","marker":"Auer et al. (2002)"},{"why":"Grounds the claim that Q-learning has good sample efficiency, supporting the paper's algorithm-ranking guidance for sample-costly system environments.","marker":"Jin et al. (2018)"},{"why":"Contributes a congestion-control entry in Table 2 that includes comparison against conventional traffic-control solutions, an example where baseline comparison exists.","marker":"Ruffy et al. (2018)"},{"why":"Demonstrates large performance gaps from different feature selections, supporting the paper's point that what is learned and which features are used matter critically.","marker":"Coons et al. (2008)"}],"fun_headline_variants":["Deep RL in systems: often a wrong tool for the job","Survey: deep RL often can't beat random search or greedy","Why deep RL often fails in system optimization (and what to check)","Deep RL hype check: simple baselines often win in systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The verdict that deep RL often fails to beat baselines depends on the papers compiled in the review's tables being representative and their reported numbers accurate; the authors note that many works did not supply enough data to reproduce results and did not independently rerun the reported experiments.","fun_headline_variants_meta":{"raw":{"variants":["Deep RL in systems: often a wrong tool for the job","Survey: deep RL often can't beat random search or greedy","Why deep RL often fails in system optimization (and what to check)","Deep RL hype check: simple baselines often win in systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2818,"prompt_tokens":892,"completion_tokens":1926,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1854}},"tokens_in":508,"tokens_out":1926,"duration_ms":13076,"temperature":1.0,"reasoning_tokens":1854,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:17:08.929270+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could falsify the central claim by conducting a systematic review with explicit inclusion criteria and reimplementing a sample of deep RL system-optimization systems on shared benchmarks; if most achieved large, reproducible gains over random search and bandit baselines and outperformed existing heuristics, the paper's conclusion that deep RL often does not beat baselines would be overturned.","supporting_citations":[{"cited_title":"Autophase: Compiler phase-ordering for hls with deep reinforcement learn- ing","cited_arxiv_id":null,"evidence_quote":"Provides the concrete evidence that random search alone achieved a 10% improvement over baseline, supporting the claim that simple search can match or beat deep RL."}],"review_version":1}