pith. sign in

arxiv: 2606.00913 · v1 · pith:Z5AJBFHWnew · submitted 2026-05-30 · 📊 stat.ML · cs.LG

Bandit Simulation for Average Reward Inference

Samya Praharaj , Chih-Yu Chang , Koulik Khamaru , Kelly W. Zhang This is my paper

Pith reviewed 2026-06-28 17:48 UTC · model grok-4.3

classification 📊 stat.ML cs.LG
keywords multi-armed banditsoff-policy evaluationconfidence intervalssimulation-based inferenceadaptive data collectionaverage reward estimation
0
0 comments X

The pith

Fitting a simulator to bandit data yields asymptotically valid confidence intervals for the mean reward of any evaluation policy, including adaptive ones.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes a method called Bandit Simulation for Inference that first estimates the parameters of a model for the reward distributions from data collected by a behavior policy. It then uses repeated simulations from this fitted model to construct confidence intervals for the average reward that would be obtained under a different target policy. This approach works for both on-policy and off-policy settings and does not rely on importance weighting. The key result is a proof that the resulting intervals achieve correct asymptotic coverage, together with experiments showing reliable performance in cases where standard off-policy estimators break down due to the dependencies created by adaptive sampling.

Core claim

BSI fits a parametric simulator of the bandit environment to observed data and propagates uncertainty in the estimated parameters through Monte Carlo simulation to produce confidence intervals for the mean reward under any chosen evaluation policy. The procedure requires only weak exploration conditions on the behavior policy and avoids reweighting. Under standard regularity conditions the intervals are asymptotically valid, meaning their coverage probability converges to the nominal level as the number of observed samples grows.

What carries the argument

The BSI simulator-fitting step, which estimates reward-distribution parameters from behavior-policy data and then resamples entire trajectories under the evaluation policy to build the interval.

If this is right

  • Valid inference becomes possible for the performance of black-box adaptive algorithms without requiring the evaluation policy to be fixed in advance.
  • Off-policy evaluation no longer needs importance weights when a simulator can be fit to the observed rewards.
  • The same fitted simulator can be reused to obtain intervals for multiple different evaluation policies.
  • Only mild conditions on how often the behavior policy tries each arm are needed for the asymptotic guarantee.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach could be extended to sequential decision problems beyond bandits by fitting a simulator of the full transition and reward dynamics.
  • If the simulator family is chosen to be nonparametric or very flexible, the method might remain valid under weaker modeling assumptions than parametric forms typically require.
  • In practice one could test the robustness of the intervals by refitting the simulator under several different parametric families and checking whether the intervals overlap substantially.

Load-bearing premise

The parametric family used for the simulator must be flexible enough that uncertainty in its fitted parameters correctly captures the remaining uncertainty about the evaluation policy's performance.

What would settle it

A simulation study in which the true reward distributions lie outside the fitted simulator class and the empirical coverage of the resulting intervals falls substantially below the nominal level.

read the original abstract

Multi-arm bandit algorithms are increasingly used in online platforms, clinical trials, and social science experiments, but valid statistical inference on their performance remains an open challenge. After deploying bandits, a natural question is whether one can construct a confidence interval for its mean reward and assess whether it reliably outperforms a baseline policy. The total reward achieved in any single bandit deployment is random, and deploying a bandit twice on the same population typically yields different reward trajectories due to stochastic rewards. Standard statistical inference methods cannot be used because bandit algorithms introduce complex dependencies in the collected data, which violate the i.i.d. assumption underlying many classical approaches. Moreover, existing inference methods for adaptively collected data only apply to estimands that do not depend on the data-collection algorithm (such as the mean reward under a fixed action). We propose Bandit Simulation for Inference (BSI), a framework that fits a simulator of the bandit environment from observed data--either on-policy or off-policy--and uses it to estimate the mean reward under any evaluation policy, including adaptive blackbox algorithms. BSI formally propagates uncertainty in the estimated simulator parameters into the confidence interval construction. Furthermore, for BSI to be valid, it requires only weak exploration assumptions on the behavior policy and avoids importance weighting. We prove that BSI yields asymptotically valid confidence intervals, and demonstrate empirically that it maintains nominal coverage in settings where standard off-policy evaluation methods fail.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes Bandit Simulation for Inference (BSI), a method that fits a parametric simulator of the bandit environment (rewards and transitions) from observed data collected under a behavior policy, then uses Monte Carlo simulation under this fitted model to estimate the mean reward of any target policy (including adaptive black-box algorithms) while propagating uncertainty in the simulator parameters to produce confidence intervals. The central claims are that BSI yields asymptotically valid CIs under only weak exploration assumptions on the behavior policy, avoids importance weighting, and empirically maintains nominal coverage in regimes where standard off-policy evaluation fails.

Significance. If the asymptotic result holds under the stated conditions, BSI would provide a practical route to valid inference on the performance of deployed bandit algorithms without requiring the evaluation policy to be fixed or the data to satisfy i.i.d. assumptions. This addresses a genuine gap in adaptive data settings common to platforms and trials. The empirical demonstration that coverage is maintained where OPE breaks is a concrete strength, though its generality depends on the simulator class.

major comments (2)
  1. [Abstract / theoretical development] The asymptotic validity proof (referenced in the abstract and presumably detailed in the theoretical section) requires that the parametric simulator family contains the true conditional reward distribution under the behavior policy; otherwise the estimated parameter covariance does not correctly capture uncertainty in the evaluation policy's mean reward. This assumption is implicit in the framework description but is not listed explicitly among the conditions for the theorem, nor is robustness to misspecification analyzed.
  2. [Assumptions and main theorem] The weak exploration assumption on the behavior policy is invoked to justify consistency of the simulator fit, but the precise rate or coverage condition needed for the parameter uncertainty to translate into valid CIs for the evaluation policy is not stated with a concrete theorem reference or counter-example when it fails.
minor comments (2)
  1. [Notation] Notation for the simulator parameters and the mapping from fitted parameters to the evaluation-policy mean reward should be introduced earlier and used consistently.
  2. [Experiments] The empirical section would benefit from an explicit statement of the data-generating process and the exact simulator class used in each experiment so that readers can assess whether the coverage results rely on correct specification.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on the theoretical assumptions. We address each point below and will revise the manuscript to improve clarity on the conditions required for asymptotic validity.

read point-by-point responses

  1. Referee: [Abstract / theoretical development] The asymptotic validity proof (referenced in the abstract and presumably detailed in the theoretical section) requires that the parametric simulator family contains the true conditional reward distribution under the behavior policy; otherwise the estimated parameter covariance does not correctly capture uncertainty in the evaluation policy's mean reward. This assumption is implicit in the framework description but is not listed explicitly among the conditions for the theorem, nor is robustness to misspecification analyzed.

    Authors: We agree that correct specification of the parametric simulator family (i.e., the true conditional distributions lie in the model class) is required for the asymptotic validity of the confidence intervals, as the delta-method or bootstrap propagation of parameter uncertainty relies on consistency and asymptotic normality of the MLE under correct specification. This is a standard assumption in parametric inference but was not stated explicitly in the theorem conditions. We will add it as an explicit assumption (e.g., Assumption on model correctness) and reference it in the main theorem. Regarding robustness to misspecification, our analysis does not cover it; we can add a brief discussion in the paper noting this as a limitation and that coverage may fail under misspecification, consistent with standard parametric results. revision: yes

  2. Referee: [Assumptions and main theorem] The weak exploration assumption on the behavior policy is invoked to justify consistency of the simulator fit, but the precise rate or coverage condition needed for the parameter uncertainty to translate into valid CIs for the evaluation policy is not stated with a concrete theorem reference or counter-example when it fails.

    Authors: The weak exploration assumption (ensuring positive probability of visiting all arms sufficiently often) is used to establish consistency of the simulator parameter estimates. The main theorem (Theorem on asymptotic validity of BSI) then combines this with standard MLE asymptotics and continuous mapping to obtain valid CIs for the evaluation policy mean reward. We will add an explicit cross-reference from the assumption to the theorem statement and clarify the required conditions (e.g., the behavior policy must ensure the information matrix is positive definite at the true parameter). A counter-example when the assumption fails (leading to inconsistent simulator parameters and invalid CIs) can be added to the appendix if space permits. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation relies on external asymptotic arguments

full rationale

The paper introduces BSI as a simulation-based procedure that fits a parametric simulator from data and propagates parameter uncertainty to CIs for policy evaluation. The central claim of asymptotic validity is presented as following from standard M-estimation and delta-method arguments under the assumption that the simulator class contains the true conditional reward law. No equations reduce a claimed prediction to a fitted quantity by construction, no load-bearing uniqueness theorems are imported via self-citation, and the method is not shown to be equivalent to its inputs. The provided abstract and reader summary contain no self-referential definitions or renamings that would trigger the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the ability to fit a simulator whose parameters carry the relevant uncertainty and on the weak exploration condition; no new physical entities are introduced.

free parameters (1)
  • simulator parameters
    Parameters of the fitted environment model are estimated from data and then resampled to propagate uncertainty; these are the quantities whose distribution supplies the confidence interval.
axioms (1)
  • domain assumption weak exploration assumptions on the behavior policy
    Stated as sufficient for validity; the precise statement appears in the framework section of the abstract.

pith-pipeline@v0.9.1-grok · 5787 in / 1288 out tokens · 19185 ms · 2026-06-28T17:48:41.254625+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 14 canonical work pages · 2 internal anchors

  1. [1]

    Simulation-basedinferenceforadaptiveexperiments

    BrianMCho,AurélienBibaut,andNathanKallus. Simulation-basedinferenceforadaptiveexperiments. arXiv preprint arXiv:2506.02881,

    work page arXiv

  2. [2]

    Learning to call: A field trial of a collaborative bandit algorithm for improved message delivery in mobile maternal health.arXiv preprint arXiv:2507.16356,

    Arpan Dasgupta, Mizhaan Maniyar, Awadhesh Srivastava, Sanat Kumar, Amrita Mahale, Aparna Hegde, Arun Suggala, Karthikeyan Shanmugam, Aparna Taneja, and Milind Tambe. Learning to call: A field trial of a collaborative bandit algorithm for improved message delivery in mobile maternal health.arXiv preprint arXiv:2507.16356,

    work page arXiv

  3. [3]

    Statistical inference under adaptive sampling with linucb.arXiv preprint arXiv:2512.00222,

    Wei Fan, Kevin Tan, and Yuting Wei. Statistical inference under adaptive sampling with linucb.arXiv preprint arXiv:2512.00222,

    work page arXiv

  4. [4]

    arXiv preprint arXiv:2509.05460 , year=

    Diego Feijer, Himan Abdollahpouri, Sanket Gupta, Alexander Clare, Yuxiao Wen, Todd Wasson, Maria Dimakopoulou, Zahra Nazari, Kyle Kretschman, and Mounia Lalmas. Calibrated recommendations with contextual bandits.arXiv preprint arXiv:2509.05460,

    work page arXiv

  5. [5]

    Statistical inference for misspecified contextual bandits.arXiv preprint arXiv:2509.06287,

    Yongyi Guo and Ziping Xu. Statistical inference for misspecified contextual bandits.arXiv preprint arXiv:2509.06287,

    work page arXiv

  6. [6]

    Stable thompson sampling: Valid inference via variance inflation.arXiv preprint arXiv:2505.23260, 2025

    Budhaditya Halder, Shubhayan Pan, and Koulik Khamaru. Stable thompson sampling: Valid inference via variance inflation, 2025.URL https://arxiv. org/abs/2505.23260. Qiyang Han, Koulik Khamaru, and Cun-Hui Zhang. Ucb algorithms for multi-armed bandits: precise regret and adaptive inference.arXiv preprint arXiv:2412.06126,

    work page arXiv 2025

  7. [7]

    Inference with the upper confidence bound algorithm

    Koulik Khamaru and Cun-Hui Zhang. Inference with the upper confidence bound algorithm.arXiv preprint arXiv:2408.04595,

    work page arXiv

  8. [8]

    Near-optimal inference in adaptive linear regression.arXiv preprint arXiv:2107.02266, 2021

    Koulik Khamaru, Yash Deshpande, Tor Lattimore, Lester Mackey, and Martin J Wainwright. Near- optimal inference in adaptive linear regression.arXiv preprint arXiv:2107.02266,

    work page arXiv

  9. [9]

    Avoiding the price of adaptivity: Inference in linear contextual bandits via stability.arXiv preprint arXiv:2512.20368,

    Samya Praharaj and Koulik Khamaru. Avoiding the price of adaptivity: Inference in linear contextual bandits via stability.arXiv preprint arXiv:2512.20368,

    work page arXiv

  10. [10]

    Efficient inference after directionally stable adaptive experiments.arXiv preprint arXiv:2602.21478,

    Zikai Shen, Houssam Zenati, Nathan Kallus, Arthur Gretton, Koulik Khamaru, and Aurélien Bibaut. Efficient inference after directionally stable adaptive experiments.arXiv preprint arXiv:2602.21478,

    work page arXiv

  11. [11]

    A review of off-policy evaluation in reinforcement learning,

    Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in reinforce- ment learning.arXiv preprint arXiv:2212.06355,

    work page arXiv

  12. [12]

    Optimism Stabilizes Thompson Sampling for Adaptive Inference

    16 Bandit Simulation for Average Reward Inference Shunxing Yan and Han Zhong. Optimism stabilizes thompson sampling for adaptive inference.arXiv preprint arXiv:2602.06014,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    Statistical inference after adaptive sampling for longitudinal data.arXiv preprint arXiv:2202.07098, 2022

    Kelly W Zhang, Lucas Janson, and Susan A Murphy. Statistical inference after adaptive sampling for longitudinal data.arXiv preprint arXiv:2202.07098,

    work page arXiv

  14. [14]

    Impatient Bandits: Optimizing for the Long-Term Without Delay

    KellyWZhang, ThomasBaldwin-McDonald, KamilCiosek, LucasMaystre, andDanielRusso. Impatient bandits: Optimizing for the long-term without delay.arXiv preprint arXiv:2501.07761,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Next, we swap the order of the derivative and integral above using the dominated convergence theorem

    =∇ 𝜆𝔼𝜆,𝜋1 [ ¯𝑅𝑇 ]=∇ 𝜆 ∑︁ 𝑎1:𝑇 ∈ A𝑇 ∫ 𝑟1:𝑇 ∈ℝ𝑇 ¯𝑟𝑇 ·ℙ 𝜆,𝜋1 (𝐴 1:𝑇 =𝑎 1:𝑇, 𝑅1:𝑇 =𝑟 1:𝑇)𝑑𝑟 1:𝑇 , for ¯𝑅𝑇 ≜ 1 𝑇 Í𝑇 𝑡=1 𝑅𝑡 and¯𝑟𝑇 ≜ 1 𝑇 Í𝑇 𝑡=1 𝑟𝑡. Next, we swap the order of the derivative and integral above using the dominated convergence theorem. = ∑︁ 𝑎1:𝑇 ∈ A𝑇 ∫ 𝑟1:𝑇 ∈ℝ𝑇 ∇𝜆 ¯𝑟𝑇 ·ℙ 𝜆,𝜋1 (𝐴 1:𝑇 =𝑎 1:𝑇, 𝑅1:𝑇 =𝑟 1:𝑇) 𝑑𝑟1:𝑇 (17) Specifically, we can apply the do...

    2004

  16. [16]

    The variance are assumed to be unknown and estimated from the data

    and unit variance. The variance are assumed to be unknown and estimated from the data. The behavior policy is uniform, and the evaluation policy is𝜖-Greedy. True parameter𝜃★ is obtained via 50,000independent MC replications of the true environment. B.2. Semi-Synthetic Environment Construction To evaluate the empirical performance of BSI in a more realisti...

    2022

  17. [17]

    The Gaussian posterior for each arm is 𝜃𝑎 | H 𝑡−1 ∼𝑁(𝑚 𝑎,𝑡−1 , 𝑣𝑎,𝑡−1 )

    for each arm, where𝑚0 is the prior mean,𝑣0 is the prior variance, and𝜎is the reward noise standard deviation. The Gaussian posterior for each arm is 𝜃𝑎 | H 𝑡−1 ∼𝑁(𝑚 𝑎,𝑡−1 , 𝑣𝑎,𝑡−1 ). Thompson Sampling selects an action a according to the posterior probability it is optimal: ˜𝜋𝑡 (𝑎| H 𝑡−1 )=ℙ(𝑎=argmax 𝑎∈ A𝜃𝑎 | H 𝑡−1 ). We further clip the action selection ...

    2021

  18. [18]

    ConfidenceIntervalviaBootstrap.Toaccountforthesamplingvariabilityintroducedbyestimating b𝑞from data, the confidence interval is constructed via a nonparametric bootstrap

    The reward model is taken to be the conjugate normal–normal posterior mean, b𝑞(𝑎)= 𝑚0/𝑣0 +𝑁 𝑎 b𝜇𝑎/𝜎2 1/𝑣0 +𝑁 𝑎/𝜎2 . ConfidenceIntervalviaBootstrap.Toaccountforthesamplingvariabilityintroducedbyestimating b𝑞from data, the confidence interval is constructed via a nonparametric bootstrap. For each replication 𝑏= 1, . . . , 𝐵, a bootstrap sample of size𝑇off i...

    2021