{"id":"80d64ac8-fad6-4bdf-a1ff-b34cb99eb215","arxiv_id":"1908.06973","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":0.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of reinforcement learning applications across seven industries, restating prior work without new results.","lead":"This paper surveys how reinforcement learning is applied across seven industries, from recommender systems to robotics. It is a broad review of prior work, not a presentation of new experiments or theory.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Survey's central claim is weakened by presenting non-RL methods (contextual bandits, MPC) as RL applications, so the evidence does not robustly support the claimed 'far-sighted' RL advantage.","rationale":"The reader classified this paper as UNVERDICTED because it is a survey or book outline rather than a research contribution; I agree with that classification. My concern is different from the reader's weakest assumption about unverified vendor-reported numbers. The more load-bearing issue for the paper's central claim is internal: the survey says RL is defined by sequential, far-sighted decision-making, but then includes contextual bandits and MPC as flagship applications. That does not make the paper a fraud or even a bad survey, but it means the evidence base is partly about methods that do not exhibit the claimed RL advantage. The correct response is not to reject the paper as a research result, since it never claimed to be one; it is to note that the survey's promotional conclusion overreaches its own examples. Thus the verdict remains UNVERDICTED, but a careful reader should treat the application summaries as covering a broader 'sequential decision-making' umbrella rather than RL proper.","tokens_in":28776,"tokens_out":3446,"duration_ms":39331,"concrete_test":"Build a method taxonomy for the featured applications in Sections 2.1 through 8.1, marking each as RL or not-RL according to three criteria: (a) the model includes state transitions; (b) the objective is cumulative or discounted return over multiple decisions; and (c) the policy is learned from interaction rather than solved by planning or supervised fitting. For Section 2.1, consult Agarwal et al. (2016) to confirm the Decision Service uses a contextual bandit with no transition model; for Section 4.1, consult Lazic et al. (2018) to confirm the cooling controller is MPC rather than RL. If two or more sections fail criteria (a)-(c), then the survey's headline claim should be qualified to 'RL and related sequential decision-making methods' rather than 'RL.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 1.10 is that RL is distinctive because it is sequential and far-sighted, optimizing long-term cumulative rewards, and that 'there is a chance for RL to help automate and optimize a manually designed strategy in a task.' For this claim to be supported, the featured case studies should actually use RL: an MDP with state transitions, multi-step credit assignment, and a learned policy. Several showcased applications do not meet this bar. Section 2.1 describes the Decision Service as a contextual bandit system: it selects actions from features with immediate rewards and has no state transitions, so it is one-shot rather than far-sighted. Section 4.1 describes data center cooling as model-predictive control (MPC), which plans over a learned linear model using short-horizon optimization, not trial-and-error policy learning. Section 5.1's option pricing methods (least-squares Monte Carlo, approximate value iteration, LSPI) are approximate dynamic programming techniques as much as RL. The paper even states in Section 1.8 that if contextual bandits are more suitable, 'RL is not the right formulation,' yet it later presents bandit systems as RL applications without flagging the distinction. This internal tension matters because the survey's positive case for RL rests on the selected applications; if a substantial fraction are not actually RL, the evidence does not support the paper's defining claim about RL's far-sightedness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a survey-style draft chapter on reinforcement learning applications. It opens with an introduction to RL concepts, a shortest-path example, a discussion of practical issues, a workflow for using RL, and study resources in Section 1. It then reviews selected applications in recommender systems (Section 2), computer systems (Section 3), energy (Section 4), finance (Section 5), healthcare (Section 6), robotics (Section 7), and transportation (Section 8). The paper's central claim is stated in Section 1.10: RL is a general, sequential, far-sighted decision-making paradigm, and there is a chance for RL to automate and optimize manually designed strategies in many tasks.","tokens_in":29037,"tokens_out":9462,"duration_ms":85583,"significance":"The survey is broad and clearly organized, with useful annotated bibliographies. It explicitly names important practical obstacles, such as the deadly triad, sample efficiency, reward misspecification, and off-policy evaluation, which helps a non-specialist reader calibrate expectations. It also collects several recent industrial case studies, including Horizon, Decision Service, Park, and AlphaGo Zero, and it points to open-source platforms and study resources. However, the manuscript offers no new technical results, no independent verification of the reported performance numbers, and no reproducibility artifacts. The most significant concern is taxonomical: some of the featured 'RL applications' are contextual bandits or model-predictive control, which the paper itself distinguishes from RL in Section 1.8. Because the central claim leans on these case studies, the scope of the evidence must be reconciled before the survey can be recommended for publication. The fix is a matter of framing and caveats rather than new experiments, so the concern is addressable in revision.","major_comments":[{"comment":"This is the main load-bearing issue. Section 1.10 defines RL as sequential and far-sighted, and Section 1.8 states that 'if contextual bandits are more suitable... RL is not the right formulation.' Nevertheless, Section 2.1 presents the Decision Service, a contextual-bandit system with no state transitions and immediate rewards, as the first recommender-system application. By the paper's own criterion, this case study is not an RL application. The survey should either extend its definition of RL to include one-step contextual decision making, or reclassify Decision Service as a neighboring paradigm; otherwise the evidence base does not support the claim that RL's far-sightedness drives the reported successes.","section":"Section 1.8 vs Section 2.1"},{"comment":"The data center cooling case study in Section 4.1 is explicitly model-predictive control: the controller learns a linear model and optimizes a trajectory over a short horizon, not trial-and-error policy learning from evaluative feedback. Similarly, Section 5.1 describes option pricing through least-squares Monte Carlo, approximate value iteration, and LSPI; the first two are standard approximate dynamic programming methods rather than RL methods that learn policies from interaction, and the section does not distinguish these cases. If the survey intends to use a broad definition that includes all sequential decision-making under uncertainty, that definition should be stated and used consistently; if not, these case studies should be moved to a separate 'related methods' section. This matters because the central claim in Section 1.10 depends on the selected applications being genuine examples of RL.","section":"Section 4.1 and Section 5.1"},{"comment":"The statement 'It is also shown that tasks with computable descriptions in computer science can be formulated as RL problems' is a strong universality claim with no citation or proof. As written, it is unsupported and overstates the scope of the cited literature. Please provide a precise reference or a formal statement of the result, or remove the sentence. This is central because the sentence is used to support the conclusion that 'the time for reinforcement learning is coming.'","section":"Section 1.10"},{"comment":"The survey repeats vendor-reported performance numbers—for example, '>25% CTR improvements' and an '18% revenue lift' for Decision Service in Section 2.1, and an A/B test result showing Horizon 'outperforms the control non-RL model' in Section 2.2—without indicating that these are company-published results subject to selection effects and conflicts of interest. If these numbers are used as evidence for RL's real-world value, the survey should either verify them against independent evaluations or explicitly label them as vendor claims and add a caution about evidence quality.","section":"Section 2.1 and Section 2.2"}],"minor_comments":[{"comment":"The heading 'An shortest path example' should read 'A shortest path example.'","section":"Section 1.5"},{"comment":"The phrase 'trail-and-error' should be 'trial-and-error.'","section":"Section 1.4"},{"comment":"In Table 2, 'curcuit design' should be 'circuit design.'","section":"Section 3.5"},{"comment":"The sentence 'a bargaining is available' is unclear; please rephrase.","section":"Section 5.2"},{"comment":"The attribution 'AI = RL + DL' to David Silver needs a citation or should be phrased as an informal hypothesis.","section":"Section 1.10"},{"comment":"The manuscript repeatedly refers readers to the author's earlier overview (Li 2017) for additional details; a brief paragraph explaining what this manuscript adds beyond Li 2017 would help address novelty and overlap concerns.","section":"Section 1.12 and Section 7.3"},{"comment":"The detailed workshop logistics (invited talks, panelists, best-paper list) read as organizational material; consider moving it to a footnote or appendix.","section":"Section 1.7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is self-described as a draft book chapter (Section 1.11), and it substantially overlaps with the author's own earlier survey Li (2017). For a research journal, the contribution is essentially pedagogical and curational. The taxonomical inconsistency in Major Comment 1 is decisive in my assessment: once the case studies are organized according to the paper's own definition of RL, the survey's positive claims can be made supportable. I therefore recommend major revision rather than rejection, as the fix is local to framing, reclassification, and caveats."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a review chapter, not a research paper. It explicitly says so in Section 1.11, and there are no new results. What it does well is give a readable map of applied RL as of 2019: short summaries of roughly thirty papers across recommender systems, computer systems, energy, finance, healthcare, robotics, and transportation, with honest annotated bibliographies. The author clearly knows the literature and the high-level summaries are accurate enough for orientation. The practical advice in Section 1.8 (formulate the MDP, prepare data, do feature engineering, pick algorithms, run experiments, deploy) is sensible and would help a practitioner starting out.\n\nThe main soft spot is the tension between the paper's central claim and its own examples. Section 1.10 says RL is distinctive because it is sequential and far-sighted, optimizing long-term cumulative rewards. But Section 2.1 presents Decision Service, a contextual bandit system with no state transitions or multi-step credit assignment, as an RL application. Section 4.1 presents data center cooling as model-predictive control, which uses a short-horizon planned trajectory, not trial-and-error policy learning. Section 5.1's option pricing methods are approximate dynamic programming as much as RL. The paper even warns in Section 1.8 that if contextual bandits are more suitable, 'RL is not the right formulation,' yet never flags that several of its featured cases are exactly that. This is an internal inconsistency that weakens the survey's thesis.\n\nThe paper also repeats vendor-reported performance numbers like '>25% CTR improvements' and '18% revenue lift' without any critical assessment. For a survey, some of this is acceptable, but if the goal is to make the case that RL is achieving spectacular real-world results, the author should at least acknowledge that some of these numbers come from systems that are not full RL. The overlap with the author's own prior survey (Li 2017) is notable but not fatal for a book chapter that is meant to be self-contained.\n\nIn proportion: these are common survey flaws, not fatal ones. The bibliography is a useful starting point, and the chapter could be fixed by sharpening the definitions of what counts as RL and adding caveats to the reported results.\n\nWould I send this to peer review? For a research venue, no, because it contains no new research and the framing problem is substantial. As a book chapter draft, it is a reasonable first pass, but I would tell the author to rework the bandit/MPC examples before showing it to a broader audience. I would not cite this survey in my own work; I would cite the primary sources it summarizes.","headline":"A serviceable survey of RL applications whose framing overreaches by counting contextual bandits and MPC as RL, worth a quick skim for the bibliography but not a research contribution.","tokens_in":29534,"tokens_out":2394,"would_cite":false,"duration_ms":27197,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning is ready to automate far-sighted decisions across industries.","keywords":["reinforcement learning","deep RL","sequential decision making","applications survey","recommender systems","robotics","healthcare","transportation"],"falsifier":"A single careful re-evaluation of a cited deployment—for instance, an A/B test comparing a deployed RL recommender against the non-RL baseline it claimed to beat, with the same data and metrics—that fails to show the reported gain would undermine the survey's case that RL reliably delivers real-world value.","tokens_in":28569,"feed_emoji":"🤖","tokens_out":4108,"duration_ms":38145,"temperature":0.7,"pith_summary":"This paper argues that reinforcement learning is a general decision-making paradigm that is especially valuable for tasks requiring sequences of decisions, because it optimizes cumulative long-term reward rather than immediate gains. It surveys successful applications in recommender systems, computer systems, energy, finance, healthcare, robotics, and transportation, and provides a practical recipe for applying RL: formulate the problem, prepare data, engineer features, choose a representation, select an algorithm, experiment, and deploy. The value of the survey, if accurate, is to show that many manually designed strategies can in principle be automated and optimized by RL, and to give practitioners a structured path for trying it.","feed_headline":"Reinforcement learning can automate far-sighted real-world decisions","feed_subtitle":"A new survey shows the same RL toolkit improving news feeds, cloud cooling, sepsis care, and robot hands.","key_machinery":"The central mechanism is the Markov decision process (MDP) formulation: an agent observes a state, selects an action, receives a scalar reward, and transitions to a next state, with the goal of maximizing expected discounted cumulative reward. The paper emphasizes that RL methods like Q-learning and policy gradients work without a model of the environment, and that a learned optimal value function encodes global, far-sighted information. This machinery lets RL improve strategies in problem domains as varied as news recommendation, device placement, data-center cooling, option pricing, sepsis treatment, robot manipulation, and order dispatching.","core_discovery":"The paper's central claim is that RL is a general, far-sighted, and promising decision-making paradigm, and that any task convertible into a sequential decision-making problem with definable states, actions, and rewards is a candidate for RL automation. The author asserts that RL has achieved major successes in games and in real products, and that the time for RL is coming: RL will play increasingly important roles in real-life applications, with both deep learning and RL prospering. The supporting evidence is a curated set of application case studies, each presented as an RL formulation with states, actions, rewards, and reported performance gains.","pith_inferences":["If the survey's evidence is representative, the limiting factor for RL adoption is not the algorithm but the availability of data and a reliable simulator; tasks without either will remain out of reach.","The survey's implicit claim that RL generalizes across such diverse domains suggests a testable hypothesis: a single family of algorithms (e.g., Q-learning variants) should reach competitive performance on a broad benchmark of sequential decision tasks drawn from several industries.","The reported performance gains are vendor- or team-reported and may not replicate under independent evaluation; a systematic, third-party replication study of the cited deployments would clarify how much of the claimed value is real."],"forward_implications":["Any manually designed policy for a sequential task with measurable outcomes can in principle be replaced or improved by an RL agent trained on sufficient data or a high-fidelity simulator.","RL's long-term view can beat one-shot, myopic supervised learning in tasks where immediate reward is a misleading guide, as shown by the shortest-path example.","Off-policy learning and counterfactual policy evaluation make it feasible to train and validate RL policies on logged data, which is crucial for healthcare and other fields where online exploration is risky.","Deploying RL in production is practical today: open-source platforms and end-to-end pipelines exist for recommender systems, cloud resource management, and notification delivery."],"supporting_citations":[{"why":"Introduces the Deep Q-Network, the foundational deep RL algorithm that shows human-level control from raw pixels.","marker":"(Mnih et al., 2015)"},{"why":"Reports AlphaGo's victory, the landmark success that motivates RL's broader promise.","marker":"(Silver et al., 2016)"},{"why":"Describes the Decision Service, the contextual bandit system that supplies the recommender-system case study.","marker":"(Agarwal et al., 2016)"},{"why":"Presents Horizon, the open-source applied RL platform used for notification and video streaming optimization.","marker":"(Gauci et al., 2019)"},{"why":"Provides the deep RL news recommendation framework DRN, which reports improvements over non-RL baselines.","marker":"(Zheng et al., 2018)"},{"why":"Shows RL learning sepsis treatment strategies from intensive-care data, supporting the healthcare application claim.","marker":"(Komorowski et al., 2018)"},{"why":"Demonstrates model-predictive control for data-center cooling, the energy-sector success story.","marker":"(Lazic et al., 2018)"},{"why":"Reports learning dexterous in-hand manipulation in simulation and transferring it to a real robot hand.","marker":"(OpenAI et al., 2018)"},{"why":"Presents the legged-robot controller that transfers from simulation to a real four-legged robot.","marker":"(Hwangbo et al., 2019)"},{"why":"Provides the ridesharing order-dispatching application that grounds the transportation chapter.","marker":"(Tang et al., 2019)"}],"fun_headline_variants":["RL: one method for far-sighted decisions in many real-world domains","Survey maps RL's reach: from recommendations to transportation","Reinforcement learning: a universal approach to sequential decision-making","From cloud cooling to sepsis care: RL's practical toolkit","How RL automates far-sighted choices across industries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the performance numbers and qualitative conclusions it repeats from the cited industry papers are accurate, since it does not independently verify any of them.","fun_headline_variants_meta":{"raw":{"variants":["RL: one method for far-sighted decisions in many real-world domains","Survey maps RL's reach: from recommendations to transportation","Reinforcement learning: a universal approach to sequential decision-making","From cloud cooling to sepsis care: RL's practical toolkit","How RL automates far-sighted choices across industries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1795,"prompt_tokens":703,"completion_tokens":1092,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":319,"completion_tokens_details":{"reasoning_tokens":1009}},"tokens_in":319,"tokens_out":1092,"duration_ms":11646,"temperature":1.0,"reasoning_tokens":1009,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:36:38.565216+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A single careful re-evaluation of a cited deployment—for instance, an A/B test comparing a deployed RL recommender against the non-RL baseline it claimed to beat, with the same data and metrics—that fails to show the reported gain would undermine the survey's case that RL reliably delivers real-world value.","supporting_citations":[{"cited_title":"A., Veness, J., Bellemare, M","cited_arxiv_id":null,"evidence_quote":"Introduces the Deep Q-Network, the foundational deep RL algorithm that shows human-level control from raw pixels."},{"cited_title":"J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al","cited_arxiv_id":null,"evidence_quote":"Reports AlphaGo's victory, the landmark success that motivates RL's broader promise."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the Decision Service, the contextual bandit system that supplies the recommender-system case study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents Horizon, the open-source applied RL platform used for notification and video streaming optimization."},{"cited_title":"J., Xie, X., and Li, Z","cited_arxiv_id":null,"evidence_quote":"Provides the deep RL news recommendation framework DRN, which reports improvements over non-RL baselines."},{"cited_title":"A., Badawi, O., Gordon, A","cited_arxiv_id":null,"evidence_quote":"Shows RL learning sepsis treatment strategies from intensive-care data, supporting the healthcare application claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates model-predictive control for data-center cooling, the energy-sector success story."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports learning dexterous in-hand manipulation in simulation and transferring it to a real robot hand."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents the legged-robot controller that transfers from simulation to a real four-legged robot."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ridesharing order-dispatching application that grounds the transportation chapter."}],"review_version":1}