Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions

Asim H. Gazi; Daiqi Gao; Kelly W. Zhang; Susan A. Murphy; Yongyi Guo; Ziping Xu

Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions

T0 review · 0 major / 7 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Real-world RL fails on two statistical hurdles: limited interaction and shifting environments, and this survey argues the field's agenda should revolve around a three-component deployment lifecycle.

desk verdict Useful survey that reorganizes statistical RL around the deployment-redeployment loop; the two-challenge premise is asserted rather than proved, but the framework earns careful reading. read the letter →

arxiv 2601.15353 v2 pith:BWLFEWNE submitted 2026-01-21 stat.AP cs.LGstat.ML

Asim H. Gazi , Yongyi Guo , Daiqi Gao , Ziping Xu , Kelly W. Zhang , Susan A. Murphy This is my paper · ORCID

classification stat.APcs.LGstat.ML

keywords reinforcementlearningstatisticalinferenceadaptiveexperimentsinterventionssequentialdeploymentscontinualoffline-to-onlineRLnonstationarity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the gap between reinforcement learning research and real-world deployment is primarily statistical, not algorithmic. It identifies two recurring obstacles: agents often cannot interact extensively with the target environment, and environments change substantially over time, requiring redesign and redeployment. To address these, the paper frames practice-oriented RL as a three-component process—online learning during deployment, offline analysis between deployments, and repeated deployment–redeployment cycles for continual improvement. A sympathetic reader would take away that single-deployment regret minimization is insufficient; the field needs methods for sample-efficient online learning, valid offline inference from adaptively collected data, and principled ways to sequence deployments. If this framing holds, it redirects research effort toward the statistical backbone of RL systems that interact with humans in high-stakes settings.

What carries the argument

The central mechanism is the three-component deployment–redeployment loop depicted in Figure 1: within-deployment online learning and optimization, between-deployment offline learning and inference, and a continual feedback process that connects the two. This loop is the organizing device that structures the survey and the research agenda. It reframes offline-to-online RL as one part of a broader cycle that also includes online-to-offline knowledge transfer and the repeated offline-online-offline improvement process. The machinery does the work of tying together otherwise separate literatures—regret minimization, off-policy evaluation, adaptive experiments, pooling algorithms, and continual

What would settle it

A concrete disproof would be a real-world RL deployment that possesses abundant interaction with a high-fidelity simulator and a stable environment yet still fails due to non-statistical causes such as safety constraints, incentive misalignment, or regulatory barriers—or, conversely, a successful deployment that requires no between-deployment offline analysis and no redeployment cycles.

Watch

Extended reading notes

Core claim

The paper's central claim is a framing: applying RL in the real world should be understood as a three-component process—(i) online learning and optimization while the algorithm is deployed, (ii) post- or between-deployment offline analysis and statistical inference, and (iii) repeated cycles of deployment and redeployment that enable continual improvement. The authors contend that two challenges drive most real-world RL failures: limited ability to interact with the target environment or a high-fidelity simulator, and substantial changes in the environment that require re-optimizing or redesigning the RL system. They survey recent statistical advances across these three components—sample-eff

Load-bearing premise

The paper's agenda rests on the premise that the two dominant obstacles to real-world RL are statistical—scarcity of interaction with the environment and substantial environmental change—and that these can be addressed within standard MDP, bandit, and POMDP abstractions.

Editorial extensions

If this is right

If the three-component framing is correct, research on RL for human-centered applications should prioritize sample efficiency and valid inference over pure regret minimization.
Offline data from prior deployments becomes a resource not just for warm-starting the next deployment but for offline algorithm selection and for enabling valid between-deployment inference.
The deployment–redeployment loop implies that the design of the RL algorithm—its exploration strategy and its model—must anticipate downstream statistical analysis, since adaptive data are non-i.i.d. and standard inference methods break down.
Pooling data across individuals can improve online learning but introduces algorithm-driven dependencies that require new statistical inference methods; the paper's survey suggests this is a core open problem.
The framework elevates continual learning and replicability to first-class concerns, alongside reward maximization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The framework implicitly calls for a new benchmark standard: evaluating RL algorithms not on a single deployment's cumulative reward but on performance across repeated deploy–analyze–redeploy cycles with distribution shift between cycles.
A testable extension is the development of 'deployment-aware' algorithms whose exploration budgets are tuned for the statistical power of downstream inference, not just for within-deployment regret.
If nonstationarity and limited interaction are the true binding constraints, then LLM-based in-context agents and causal-DAG-informed priors may prove more impactful than regret-optimal algorithms—an avenue the paper leaves largely empirical.
The survey's emphasis on 'replicability' suggests a new statistical property for RL algorithms—convergence of action-selection probabilities—that practitioners could adopt as a pre-deployment diagnostic using offline simulators.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Plain take: the three-component framing — online learning during deployment, offline inference between deployments, and the deployment-redeployment loop — is a real synthesis, not just an abstract rearrangement. Prior surveys mostly split offline RL and online RL; this one points at the interface between them, and that points at a genuine gap in the literature.

It does several things well. The examples (HeartSteps, adaptive experiments, oral health) are concrete and appropriately varied. The methodological review is broad and, as far as I can tell, accurate. The paper is unusually honest about its own limitations: it explicitly says causal-DAG RL has not been deployed with humans, that LLM in-context agents have not been evaluated in a real online decision loop, and that statistical inference for pooled-data adaptive interventions is largely open. Section 5.1, on inference after pooling algorithms, is one of the most useful parts; that is a hard problem that deserves more attention.

The soft spots are real but not disqualifying. The central premise — that limited interaction and environmental change are the two recurring obstacles — is asserted through a narrative selection of health/education examples rather than a systematic accounting of where real-world RL fails. The stress-test note is right that safety, trust, incentive alignment, and pre-specification constraints are acknowledged but not built into the framework. So this is a statistical perspective on real-world RL, not a demonstration that statistical obstacles are the dominant ones. The hedging ('often,' 'we find') keeps it from being wrong, but it does mean the framework's centrality is a claim about emphasis, not an empirical finding. The heavy self-citation is visible in the open-problems sections, but the framework is also applied to plenty of external work, so I don't see it as load-bearing. One sentence to check before publication: the claim that no prior review highlights the offline-to-online-to-offline process is stronger than I would want to defend.

Who it's for: statisticians and applied RL researchers working on health/education interventions, and anyone looking for a map of open problems in sequential-deployment RL. It deserves a serious referee. My recommendation: send it out; ask for an extra paragraph acknowledging non-statistical constraints as co-equal in some deployments, and soften the 'no prior review' sentence.

Referee Report

0 major / 7 minor

Summary. The paper proposes that real-world reinforcement learning, especially in human-facing applications, is best understood through two statistical obstacles—limited opportunity for extensive interaction with the target environment, and substantial environmental change across deployments—and organizes the field around a three-component process: within-deployment online learning and optimization, between-deployment offline analysis, and repeated deployment–redeployment cycles for continual improvement. It then reviews recent work relevant to each component, including sample-efficient online learning, bias–variance tradeoffs, causal RL, LLM-based decision making, inference from adaptively collected data, offline-to-online initialization, and continual learning, and identifies open problems for future research.

Significance. If the framework is adopted, it provides a useful organizing device for statistical RL research that goes beyond single-deployment regret minimization and highlights under-emphasized problems such as online-to-offline inference, replicability under misspecification, and tradeoffs across sequential deployments. The paper is careful in marking where evidence is absent: for example, Sec. 4.1 states that no real-world human deployment of causal-DAG-based online RL is known, and Sec. 4.2 states that LLM in-context agents have not yet been evaluated in a real-world online sequential decision loop. These explicit limitations are a strength of the paper. The main limitation is that the centrality of the two statistical challenges is asserted through a narrative selection of examples rather than a systematic survey; the authors acknowledge this in Sec. 1 and qualify the claim with “often,” so I do not treat this as an internal inconsistency. Overall, the paper is a fair-minded, use-inspired survey that should stimulate discussion.

minor comments (7)

[Sec. 2.1] In the contextual bandit discussion, the optimal policy is written as π* ∈ argmax_π Eπ[R|S=s]; this should be argmax_a E[R|S=s, A=a] (or the argmax over policies should be defined by their induced action distributions). As written, the expression is not quite correct.
[Sec. 2.1] The sentence “the reward R_t is a known, possibly stochastic, function of (S_t, A_t, S_{t+1})” should be “unknown” (or “modeled”) rather than “known”; the reward model is part of the environment that the agent must learn. This is a typographical but potentially confusing error.
[Sec. 5.2] The sentence “All of the above-mentioned simulator-based and offline-to-online approaches share a common limitation: they generate next states and rewards based only on the current state–action pair” conflates two distinct classes. Offline-to-online initialization methods do not generate next states and rewards; they initialize or regularize policies/value functions. The intended point about stationarity/Markov assumptions should be reworded to apply to the simulator-based methods explicitly and to offline-to-online methods only insofar as they rely on the same underlying data assumptions.
[Sec. 4.1] The discussion of discount-factor regularization would benefit from a reminder that smaller γ also changes the optimization objective, not only the variance of return estimates. The current wording is acceptable but risks being read as a pure variance-reduction technique.
[Sec. 5.2] There is a typo: “conditions clikely” should be “conditions likely.”
[Sec. 1 / Abstract] The paper might explicitly separate “statistical challenges” from broader deployment challenges such as safety, human trust, and incentive alignment. The text does mention safety guardrails and prespecification requirements, but integrating these into the framing would prevent readers from over-reading the two-challenge claim as exclusive.
[Sec. 6.2] The sentence “developing algorithms that adapt over time (not just agents that can adapt over time via online RL)” is somewhat unclear; the distinction between algorithm-level and agent-level adaptation would benefit from one illustrative example.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper asserts rather than derives its framing; self-citations are frequent but not load-bearing.

full rationale

This is a narrative survey and perspective, not a derivation. The central contribution is a framing: RL in practice is organized as a three-component process (within-deployment online learning, between-deployment offline analysis, and sequential deployment-redeployment), motivated by two asserted challenges (limited interaction and environmental change). No equation is derived, no parameter is fitted and then called a prediction, and no known result is renamed as a new derivation. The paper explicitly labels itself a non-systematic survey and states that its inclusion criteria are relevance to future opportunities and addressing the two challenges ('our criteria for including literature in our review include relevance to future opportunities and addressing the two challenges presented earlier,' Sec. 1). This is a survey-scope limitation—the narrative is self-consistent by construction—but it is not a circular derivation in the sense of the requested patterns. The paper also candidly flags missing validation for its own suggested directions (e.g., 'To our knowledge, there has not been a real-world deployment involving interactions with humans of an online RL algorithm that leverages causal DAGs,' Sec. 4.1; and 'this exact formulation... has not yet been evaluated in a real-world study,' Sec. 4.2), which further shows the authors are not presenting those directions as established results. Self-citations are numerous (e.g., Gazi et al. 2025c; Zhang et al. 2022a, 2025b; Guo and Xu 2025; Gao et al. 2025a,b; Xu et al. 2024b), but they function as illustrations of open problems or as pointers to specific theorems and deployments, not as the justification for the three-component framework itself. The empirical motivation is carried substantially by external work (DIAMANTE, REINFORCE, HeartSteps, PEARL, and others). No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via self-citation. The central claim is an organizing perspective rather than a derived result; thus the classic circularity failure modes are absent. The heavy self-citation is a minor concern about framing and emphasis, not a load-bearing circular step. Score 1 reflects the absence of any exhibited reduction of a result to its own inputs, while acknowledging the self-referential survey-design choice.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities appear because the paper makes no quantitative claims or model proposals. The framework rests on stated domain assumptions about what makes real-world RL hard and on the adequacy of standard MDP/bandit abstractions; the subjective literature selection is acknowledged.

assumptions (3)

domain assumption Real-world RL practice can be decomposed into three components: within-deployment online learning, between-deployment offline analysis, and repeated deployment–redeployment cycles.
This is the paper's organizing framework (Sec. 1, Fig. 1); it is a perspective rather than a theorem and shapes which literature is included.
domain assumption The two central obstacles are (1) limited ability to interact with the target environment and (2) substantial environmental change over time.
Stated in the abstract and Sec. 1; the review's selection and open-problem list depend on these being the most important gaps.
standard math Standard sequential-decision models (contextual bandits, MDPs, POMDPs) with finite action sets, bounded rewards, and history-dependent transitions adequately represent the deployment settings discussed.
Sec. 2 introduces these as background; they are conventional modeling assumptions, not proven here.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions." pith.science (2026). https://pith.science/paper/BWLFEWNE

@misc{pith2026260115353,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BWLFEWNE}},
  note         = {Machine review of arXiv:2601.15353}
}

read the original abstract

Reinforcement learning (RL) has achieved remarkable success in real-world decision-making across diverse domains, including gaming, robotics, online advertising, public health, and natural language processing. Despite these advances, a substantial gap remains between RL research and its deployment in many practical settings. Two recurring challenges often underlie this gap. First, many settings offer limited opportunity for the agent to interact extensively with the target environment due to practical constraints. Second, many target environments often undergo substantial changes, requiring redesign and redeployment of RL systems (e.g., advancements in science and technology that change the landscape of healthcare delivery). Addressing these challenges and bridging the gap between basic research and application requires theory and methodology that directly inform the design, implementation, and continual improvement of RL systems in real-world settings. In this paper, we frame the application of RL in practice as a three-component process: (i) online learning and optimization during deployment, (ii) post- or between-deployment offline analyses, and (iii) repeated cycles of deployment and redeployment to continually improve the RL system. We provide a narrative review of recent advances that address the statistical challenges arising across these three components, including methods for enhancing sample efficiency during online deployment, maximizing data utility for post- or between-deployment inference, and designing sequences of deployments for continual improvement. We also outline future research directions in RL that are use-inspired -- aiming for impactful application of RL in practice.

Figures

Figures reproduced from arXiv: 2601.15353 by the authors.

**Figure 2.** Illustrative causal diagrams of sequential decision-making environments: (a) contextual [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** An example of a causal DAG that describes the HeartSteps environment. HeartSteps is a [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Pooling algorithm. Depiction of how a pooling algorithm combines or “pools" data across individuals to form and update the policies over time. Adapted from Zhang et al. [2024]. episodes as each H length episode concerns a different unit sampled from the population. How…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

A Diffusion-Model Subpopulation Digital Twin for Mobile Health Deployment: A Case Study on the HeartSteps Intervention
cs.LG 2026-07 conditional novelty 6.0 of 10

A pre-train/fine-tune/calibrate diffusion-model pipeline produces subpopulation digital twins that out-reproduce simpler simulators on temporal and between-participant structure in a HeartSteps replay.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[2024]

it felt more real

ISSN 17515823. doi:10.1111/INSR.12583;PAGE:STRING:ARTICLE/CHAPTER. URL /doi/pdf/10.1111/insr.12583https://onlinelibrary.wiley.com/doi/abs/10.1111/insr. 12583https://onlinelibrary.wiley.com/doi/10.1111/insr.12583. Zhihong Deng, Jing Jiang, Guodong Long, and Chengqi Zhang. Causal reinforcement learning: A survey.Transactions on Machine Learning Research, 20...

arXiv 2023
[2025]

doi:10.1609/AAAI.V39I28.35143

ISSN 2374-3468. doi:10.1609/AAAI.V39I28.35143. URL https://ojs.aaai.org/index. php/AAAI/article/view/35143. 39 Nilesh Tripuraneni, Lee Richardson, Alexander D’Amour, Jacopo Soriano, and Steve Yadlowsky. Choosing a proxy metric from past experiments. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5803–5812, 20...

arXiv 2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2024]

it felt more real

ISSN 17515823. doi:10.1111/INSR.12583;PAGE:STRING:ARTICLE/CHAPTER. URL /doi/pdf/10.1111/insr.12583https://onlinelibrary.wiley.com/doi/abs/10.1111/insr. 12583https://onlinelibrary.wiley.com/doi/10.1111/insr.12583. Zhihong Deng, Jing Jiang, Guodong Long, and Chengqi Zhang. Causal reinforcement learning: A survey.Transactions on Machine Learning Research, 20...

arXiv 2023

[2] [2025]

doi:10.1609/AAAI.V39I28.35143

ISSN 2374-3468. doi:10.1609/AAAI.V39I28.35143. URL https://ojs.aaai.org/index. php/AAAI/article/view/35143. 39 Nilesh Tripuraneni, Lee Richardson, Alexander D’Amour, Jacopo Soriano, and Steve Yadlowsky. Choosing a proxy metric from past experiments. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5803–5812, 20...

arXiv 2024