Pith. sign in

REVIEW 4 major objections 5 minor 38 references

The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportability

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that a model-based algorithm learns an ε-optimal policy under hidden agent types and population shift with Õ(1/ε²) samples, matching the optimal dependence on ε up to logarithms.

desk verdict A coherent online extension of offline strategic MDPs with a strong realizability assumption that goes unverified in concrete examples; deserves referee time but should be read as conditional. read the letter →

arxiv 2506.09940 v1 pith:6BQKAZ6K submitted 2025-06-11 cs.LG cs.AIstat.ML

Jiachen Hu , Rui Ai , Han Zhong , Xiaoyu Chen , Liwei Wang , Zhaoran Wang , Zhuoran Yang This is my paper
classification cs.LGcs.AIstat.ML MSC 68Q3290C40
keywords onlinestrategicinteractionmodelinformationasymmetryknowledgetransportabilitynonparametricinstrumentalvariablesamplecomplexityconfoundedreinforcementlearningdistributionalEluderdimensionoptimisticplanning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether an online learner can identify a payoff-relevant system when the agents it interacts with conceal private types that confound both rewards and transitions, and can afterwards transfer that knowledge to a different population of agents for which it must optimize. It answers yes to both, proving that the model-based algorithm OPME learns an $\epsilon$-optimal policy for the target population with $\widetilde{O}(\sum_{h=1}^H B^2 d_{V,h} \tau_h C^f_h \log(|R\times P\times G\times F|/\delta)\epsilon^{-2})$ samples, with probability at least $1-\delta$ — a rate of $\widetilde{O}(1/\epsilon^2)$, matching the known episodic-RL lower bound in $\epsilon$ up to logarithms. The combination the paper treats — unobserved confounders created by strategic behavior, and a target population different from the population producing the online data — is exactly what the earlier offline analysis of strategic MDPs did not handle, and it is the combination faced in contract design, admissions, and experimental design with imperfect proxies for human subjects. The theorem also isolates the price of each difficulty: the ill-posedness measure $\tau_h$ and the knowledge-transfer (concentrability) term $C^f_h$ appear as linear multiplicative factors, so the bound states concretely how much harder a confounded, shifted problem is than an ordinary one.

What carries the argument

The carrying mechanism is a nonparametric instrumental-variable (NPIV) formulation of the confounded system. Because the endogenous noise $\xi_h$ correlates with the agent's private type, the observable conditional mean $E[r_h | s_h, a_h, e_h]$ no longer equals the true reward $R^*_h$, but the principal's own state-action pair $(s_h, a_h)$ is a valid instrument: it satisfies the conditional moment equation $E[r_h - R^*_h(s_h, a_h, e_h) | s_h, a_h] = 0$, and an identical equation holds for transitions against value functions. Estimation is done by a minimax empirical risk (Fenchel–Rockafellar dual) over a discriminator class $F_h$, and the transition dynamics are estimated by value-targeted regression against a second class $G_h$, which together contribute the $\log(|R\times P\times G\times F|/\delta)$ factor. Three coefficients then carry the analysis: the ill-posedness measure $\tau_h$ converts projected mean-square error (what the instrument equations bound) into full mean-square error (what regret needs), the knowledge-transfer term $C^f_h$ converts source-distribution error into target-distribution error, and the distributional Eluder dimension $d_{V,h}$ of the residual function classes converts per-episode estimation error into total regret via an optimism-in-the-face-of-uncertainty argument; the non-i.i.d. data produced by adaptive exploration is handled by martingale (Freedman) concentration rather than i.i.d. bounds.

What would settle it

Set up a one-step, two-state instance where the instrument $(s,a)$ has two values, the discriminator class $F_h$ contains only constant functions, and the true residual $\nu_h = R_h - R^*_h$ has different conditional expectations at the two instrument values. Then the projected mean-square error is zero while the full mean-square error is bounded below — the confidence set can hold a wrong model at zero risk, yet the value difference is positive — so running OPME on this instance and finding its returned policy worse than $\epsilon$-optimal would demonstrate that Assumption 5.1's second bullet is doing the work the theorem needs.

Watch

Extended reading notes

Core claim

The paper's claim is that information asymmetry and knowledge transportability can be handled together in online reinforcement learning without paying more than the optimal $\epsilon^{-2}$ price. Under the realizability conditions of Assumption 5.1 — the true reward and transition functions lie in the model classes, the discriminator class $F_h$ contains the instrument-projections of all estimation residuals, and $G_h$ contains the optimal value functions of all aggregated models — the OPME algorithm learns an $\epsilon$-optimal policy of the target population's aggregated model with $\widetilde{O}(\sum_{h=1}^H B^2 d_{V,h} \tau_h C^f_h \log(|R\times P\times G\times F|/\delta)\epsilon^{-2})$ samples with probability at least $1-\delta$, and the dynamical-transition variant OPME-D carries the same guarantee with $d_{M,h}$ in place of $d_{V,h}$. Since even a confounder-free episodic MDP requires $\Omega(\epsilon^{-2}\log(1/\delta))$ samples, the inverse-quadratic dependence on $\epsilon$ is optimal up to logarithms (Remark 5.6).

Load-bearing premise

The load-bearing premise is that the discriminator class $F_h$ contains, exactly, the projection of every estimation residual onto the instrument space: for every candidate error $\nu_h$ (a reward residual or a transition residual against any value function in $G$), the conditional expectation $E[\nu_h(s,a,e)|s,a]$ must be a member of $F_h$. This is a strong function-approximation condition that cannot be verified from data, and the confidence sets only control projected error, so without it the ill-posedness step cannot deliver the full mean-square error the regret bound needs.

Editorial extensions

If this is right

  • The guaranteed rate makes settings like CEO-compensation contract design, college admissions, and A/B testing with LLM stand-ins for human subjects formally solvable by adaptive experimentation at the same inverse-quadratic sample rate as ordinary episodic reinforcement learning.
  • Because $\tau_h$ and $C^f_h$ enter the bound linearly, the theorem states quantitatively how hardness decomposes: an instrument that barely identifies the residual inflates episodes by $\tau_h$, and a target population poorly covered by the source inflates them by $C^f_h$.
  • For linear MDPs with a known feature map, the distributional Eluder dimension is $\widetilde{O}(d)$, so the bound becomes the explicit $\widetilde{O}(\sum_{h=1}^H B^2 d\, \tau_h C^f_h \log(|R\times P\times G\times F|/\delta)\epsilon^{-2})$ sample count for that class.
  • The same template — instrumental-variable identification for estimation, optimistic planning for exploration, and aggregation under the target distribution for transfer — applies to any function classes satisfying the realizability and bounded-capacity conditions, so the result is not tied to a tabular or linear structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The concentrability factor $C^f_h$ yields a testable prediction the authors do not state: holding the source population fixed, the episode budget OPME needs should grow linearly as the target residual distribution diverges from the source's, and a target with support outside the source's support should be unlearnable; a synthetic linear-MDP experiment varying target-type distributions could check
  • The paper leaves open whether the problem-dependent factors $\tau_h$ and $C^f_h$ are themselves unavoidable (Remark 5.6); a matching lower bound, if it exists, would show the algorithm is tight term-by-term rather than only in $\epsilon$.
  • The authors flag that the target distribution $P^t$ and the feedback-manipulation distribution $F$ must be known to the principal; the natural next extension is to estimate $P^t$ from passive target-population samples, and the extra error of that estimation step would presumably reappear inside $C^f_h$ and multiply the bound again.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies an online episodic MDP in which, at each step h, a myopic agent with an unobserved private type t_h chooses a strategic action and provides a manipulated feedback e_h; the principal observes (s_h,a_h,e_h,r_h,s_{h+1}) but not t_h. The reward noise is endogenous (correlated with t_h), so standard regression fails, and data are collected under a source population P^s while the goal is to find an epsilon-optimal policy in the aggregated target MDP under P^t. The authors propose two model-based optimistic algorithms (OPME-G for general transition classes and OPME-D for dynamical systems), using nonparametric instrumental variable estimators with discriminator classes F and G to construct confidence sets, and transfer knowledge by plugging the estimated model into the target aggregation. The main theorem (Theorem 5.4) claims that under a realizability assumption (Assumption 5.1), OPME-G learns an epsilon-optimal policy with sample complexity O(sum_h B^2 d_{V,h} tau_h C^f_h log(|R x P x G x F|/delta) epsilon^{-2}), matching the epsilon^{-2} lower bound up to logarithms.

Significance. If the proof is repaired, the paper makes a useful contribution: it defines a natural online version of the strategic MDP with unobserved types, replaces the i.i.d. assumption of Yu et al. (2022) by a martingale analysis, adds a knowledge-transfer term C^f_h that quantifies distribution shift, and achieves the information-theoretically optimal dependence on epsilon. The identification via conditional moment equations and the use of the distributional Eluder dimension for online exploration are appropriate tools. The bound is parameterized by transparent quantities (ill-posedness tau_h, concentrability C^f_h, and distributional Eluder dimension d_{V,h}), and the paper explicitly discusses the necessity of each. However, the current proof has several gaps that must be fixed before the theorem is established; the most serious is the mismatch between the L1-type regret terms in Lemma H.6 and the first-moment bound supplied by Lemma G.3.

major comments (4)
  1. [Lemma H.6 and Theorem H.7] In Lemma H.6 the regret is bounded by sums of E_{d^{t,pi^k}_h}[|R^k_h-R^*_h|] and E_{d^{t,pi^k}_h}[|P^k_h Vbar^{pi^k}_{Mbar^k,h+1}-P^*_h Vbar^{pi^k}_{Mbar^k,h+1}|], after Jensen's inequality moves the absolute value inside the expectation. In Theorem H.7 these sums are then bounded by invoking Lemma G.3 (Lemma 41 of Jin et al. 2021), whose conclusion is of the form sum_t |E_{mu^t}[phi_t]| <= O(sqrt(dim beta K) + ...). Since |E phi| <= E |phi|, a bound on the latter does not imply the required bound on the former. The standard repair is to apply the simulation lemma without Jensen's inequality, so that the terms appearing in the regret are |E[phi_k]| (or E[phi_k] with a sign), which is exactly the quantity controlled by Lemma G.3. As written, the chain of inequalities in Lemma H.6 goes in the wrong direction for the Eluder step.
  2. [Assumption 5.1 and Lemma H.4] The second bullet of Assumption 5.1 is load-bearing: Lemma H.4 equates the projected mean-square error with l^k_h(nu_h, f[nu_h]) and needs f[nu_h] in F_h (Eqs. (H.5)-(H.6)) to convert the confidence-set event that hat L^k_h(R_h) <= beta into a bound on that projected error. If F_h omits any of these conditional projections, the max over F_h in hat L can be small while the projected MSE entering Lemma H.5 is large, and the claimed O(epsilon^{-2}) sample complexity is not supported. The assumption is not verified in Example 5.5, which specifies R_h and P_h but does not define F_h and G_h or argue that they satisfy the required closure and value-realizability conditions. The example should be completed, or the theorem should state explicitly that its applicability requires this non-verifiable closure property.
  3. [Proof of Lemma H.4] In the second inequality of the proof of Lemma H.4, the quantity hat l^k_h(nu_h, f_h) is expanded as hat l^k_h(R_h, f_h) - hat l^k_h(R^*_h, f_h). The confidence-set condition bounds the first term by beta + 1/2 ||f||^2_{2,k}, but the second term is not zero: hat l^k_h(R^*_h, f_h) = -sum_j f_h(s^j_h,a^j_h) xi^j_h. The displayed bound therefore omits a martingale term that must be controlled separately, for instance by applying Lemma H.2 to nu_h = 0. As written, this step of the proof is incomplete.
  4. [Section D.1 and Lemma H.6 (OPME-D)] Equation (D.1) defines the transition noise as eta_h = corr(t_h)+eta'_h, which makes the distribution of s_{h+1} conditional on (s_h,a_h,e_h,t_h) a Gaussian with mean G^*_h + corr(t_h) and covariance I. Lemma H.6 nevertheless states that 'by Equation (D.1)' the kernel P_h(·|s,a,e,t_h) is Gaussian with mean G_h(s,a,e) and identity covariance, and the subsequent bound drops corr(t_h). If corr(t_h) is non-constant, the true marginal transition is a Gaussian mixture and the class P_h = {G_h : N(G_h,I)} may not contain it; if corr(t_h) is a constant, that should be stated explicitly so that it can be absorbed into G^*. The OPME-D analysis needs clarification or repair, although this does not by itself invalidate the OPME-G bound in Theorem 5.4.
minor comments (5)
  1. [Section 1] The phrase 'casual inference' should be 'causal inference'.
  2. [Equation (4.6)] The summation index in the definition of hat l^k_h appears as 'kX t=tau'; it should be sum_{tau=1}^k.
  3. [Equation (4.2)] The expression E_{e_h}[r_h - R_h(s_h,a_h,e_h)|s_h,a_h] uses r_h inside the conditional expectation without making the randomness of xi_h explicit; write E[r_h|s_h,a_h]-E[R_h(s_h,a_h,e_h)|s_h,a_h] or define the expectation over e_h and xi_h.
  4. [Example 5.5] The statement 'd_{V,h} ≲ \tilde O(d)' is redundant; 'd_{V,h} = \tilde O(d)' would be clearer.
  5. [Algorithm 2] The pseudocode refers to OPME-D and OPME-G for the transition risk but does not show the definition of hat L for the transition case in the main text; explicitly point the reader to Equations (4.6) and (D.4).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the sample-complexity bound follows from stated realizability and complexity assumptions, with the epsilon-dependence benchmarked against an external lower bound.

full rationale

The derivation is self-contained in the relevant sense. The main theorem (Theorem 5.4) is an upper bound conditional on Assumption 5.1, which states realizability of R* and P*, the projection property f[nu_h] in F_h for residuals nu_h, and value-function realizability of G_h; these are assumptions, not conclusions, and none of them asserts the sample-complexity rate. The proof chain (Lemmas H.1–H.3, H.4, H.5, H.6, Theorem H.7) bounds projected mean square error via the confidence set, converts projected error to full mean square error using the problem-dependent ill-posedness measure tau_h and the concentrability term C^f_h (Definitions 5.2 and 5.3), and then applies the distributional Eluder dimension from Section E. The quantities tau_h and C^f_h are population-level definitions involving the true model and the function classes; they are not fitted from data, so the bound is not a prediction of fitted values. The dependence on epsilon is checked against the external minimax lower bound of Domingues et al. (2021), and the paper explicitly acknowledges that C^f_h is the standard concentrability notion. The cited prior work Yu et al. (2022), which shares an author, motivates the NPIV construction and the strategic MDP model, but the present paper supplies its own causal identification and martingale concentration analysis; the citation is not used as an unverified theorem that forces the conclusion. The F_h-projectability condition in Assumption 5.1 is strong and may be difficult to verify in practice, but that is an assumption-strength and correctness concern, not a circular step.

Assumptions & free parameters 0 free parameters · 7 assumptions · 0 invented entities

The central claim rests on strong realizability, instrument validity, and the assumption that the target type distribution and feedback manipulation distribution are known. These are standard for the NPIV and principal-agent literature, but they are not derived in the paper and would be hard to verify in practice.

assumptions (7)
  • domain assumption Assumption 5.1 realizability: R*_h in R_h, P*_h in P_h for all h; F_h contains the projections of residuals; G_h contains all aggregated-model optimal value functions.
    Necessary for the minimax confidence sets to control projected mean square error and for the regret decomposition to use value functions in G.
  • domain assumption The endogenous reward noise xi_h is independent of (s_h,a_h) and has zero conditional mean under the source distribution: E[xi_h|s_h,a_h]=0 (Section 3.2 and Equation (C.1)).
    Makes (s_h,a_h) a valid instrumental variable for the reward equation; used in Equation (4.1).
  • domain assumption The state-action pair (s_h,a_h) satisfies the conditional moment restrictions in Equations (4.1) and (4.5) for rewards and transitions.
    This is the instrument validity assumption that underlies the NPIV identification.
  • domain assumption The target type distribution P^t and the feedback manipulation distribution F are known to the principal (Section 6).
    The aggregated model in Equations (3.1)-(3.2) and the exploration objective in Equation (4.9) require these distributions to compute expectations.
  • standard math Function classes R, P, F, G are finite for the analysis, with covering numbers as the infinite generalization (start of Appendix H).
    Simplifies the union bound over function classes; standard in RL theory.
  • domain assumption The ill-posedness measure tau_h and the knowledge transfer multiplier C^f_h are finite (Definitions 5.2 and 5.3).
    Without these finiteness conditions, the bound diverges and knowledge transfer is impossible.
  • domain assumption All rewards, transitions, and function class members are bounded by B (Assumption 5.1).
    Used in Freedman's inequality and the confidence level choices in Equation (H.4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportability." pith.science (2026). https://pith.science/paper/6BQKAZ6K

@misc{pith2026250609940,
  author       = {Pith},
  title        = {Pith review of: The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6BQKAZ6K}},
  note         = {Machine review of arXiv:2506.09940}
}
abstract

Information asymmetry is a pervasive feature of multi-agent systems, especially evident in economics and social sciences. In these settings, agents tailor their actions based on private information to maximize their rewards. These strategic behaviors often introduce complexities due to confounding variables. Simultaneously, knowledge transportability poses another significant challenge, arising from the difficulties of conducting experiments in target environments. It requires transferring knowledge from environments where empirical data is more readily available. Against these backdrops, this paper explores a fundamental question in online learning: Can we employ non-i.i.d. actions to learn about confounders even when requiring knowledge transfer? We present a sample-efficient algorithm designed to accurately identify system dynamics under information asymmetry and to navigate the challenges of knowledge transfer effectively in reinforcement learning, framed within an online strategic interaction model. Our method provably achieves learning of an $\epsilon$-optimal policy with a tight sample complexity of $O(1/\epsilon^2)$.

Figures

Figures reproduced from arXiv: 2506.09940 by the authors.

Figure 1
Figure 1. Timeline of the interaction. rh and sh+1 (blue) are influenced by observable sh, ah, eh (red) and unobservable th (green). We use numbers to indicate the sequence of events. Conlisk, 1996). Therefore, experiment designers need to combine known human characteristics say P t with experi￾mental data derived from LLM features say P s to develop optimal mechanisms tailored for human agents. 3.2. Planning in the Online St… view at source ↗
Figure 2
Figure 2. The causal graph for the strategic interaction between the principal and the h-th agent. The red line denotes the confounding between (rh, sh+1) and th. ”IV” means instrumental variables. Sections D and H for details on hyperparameter selection and additional information on another variant OPME-D. It is challenging to estimate R∗ h and P ∗ h from the model class because of the confounding issue. We take R∗ h as an e… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 26 canonical work pages

  1. [2]

    The causal structure of the transition sh+1 is the same asr h

    To better illustrate this strategic interaction process, we take the reward rh as an example. The causal structure of the transition sh+1 is the same asr h. The reward equals rh =R ∗ h(sh, ah, eh) +ξ h by definition, whereξ h is an endogenous noise that may be confounded witht h (Yu et al., 2022; Harris et al., 2022). Definition C.1(Informal, see Pearl (2...

    work page 2009

  2. [4]

    Sequential Information Design: Learning to Persuade in the Dark

    Bernasconi, M., Castiglioni, M., Marchesi, A., Gatti, N., and Trov`o, F. Sequential information design: Learning to persuade in the dark.arXiv preprint arXiv:2209.03927,

    work page Pith review arXiv

  3. [9]

    J., Yuan, A., Gu, Q., and Jordan, M

    10 The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportability Chen, Z., Li, C. J., Yuan, A., Gu, Q., and Jordan, M. I. A general framework for sample-efficient function ap- proximation in reinforcement learning.arXiv preprint arXiv:2209.15634,

    arXiv

  4. [11]

    Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximation

    Dubey, A. and Pentland, A. Provably efficient cooperative multi-agent reinforcement learning with function approx- imation.arXiv preprint arXiv:2103.04972,

    work page Pith review arXiv

  5. [12]

    J., Kakade, S

    Foster, D. J., Kakade, S. M., Qian, J., and Rakhlin, A. The statistical complexity of interactive decision mak- ing.arXiv preprint arXiv:2112.13487, 2021a. Foster, D. J., Krishnamurthy, A., Simchi-Levi, D., and Xu, Y . Offline reinforcement learning: Fundamental barri- ers for value function approximation.arXiv preprint arXiv:2111.10919, 2021b. Futoma, J....

    arXiv 2001

  6. [13]

    and Goldberg, Y

    Gamrian, S. and Goldberg, Y . Transfer learning for related reinforcement learning tasks via image-to-image transla- tion. InInternational conference on machine learning, pp. 2063–2072. PMLR,

    work page

  7. [14]

    Generalized Principal-Agency: Contracts, Information, Games and Beyond

    Gan, J., Han, M., Wu, J., and Xu, H. Optimal coordination in generalized principal-agent problems: A revisit and extensions.arXiv preprint arXiv:2209.01146, 2022a. Gan, J., Majumdar, R., Radanovic, G., and Singla, A. Bayesian persuasion in sequential decision-making. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 5025–5...

    work page Pith review arXiv

  8. [15]

    J., and Liut, M

    Kumar, H., Musabirov, I., Reza, M., Shi, J., Kuzminykh, A., Williams, J. J., and Liut, M. Impact of guidance and interaction strategies for llm use on learner performance and perception.arXiv preprint arXiv:2310.13712,

    arXiv

Show all 38 references
  1. [17]

    Littman, M. L. Markov games as a framework for multi- agent reinforcement learning. InMachine learning pro- ceedings 1994, pp. 157–163. Elsevier,

    1994

  2. [22]

    and Jiang, N

    Nair, Y . and Jiang, N. A spectral approach to off-policy evaluation for pomdps.arXiv preprint arXiv:2109.10502,

    arXiv

  3. [24]

    Future- dependent value-based off-policy evaluation in pomdps

    Uehara, M., Kiyohara, H., Bennett, A., Chernozhukov, V ., Jiang, N., Kallus, N., Shi, C., and Sun, W. Future- dependent value-based off-policy evaluation in pomdps. arXiv preprint arXiv:2207.13081, 2022a. Uehara, M., Sekhari, A., Lee, J. D., Kallus, N., and Sun, W. Provably ef...

    arXiv

  4. [25]

    Distributed bandit learning: Near-optimal regret with efficient communica- tion.arXiv preprint arXiv:1904.06309,

    Wang, Y ., Hu, J., Chen, X., and Wang, L. Distributed bandit learning: Near-optimal regret with efficient communica- tion.arXiv preprint arXiv:1904.06309,

    1904 arXiv

  5. [26]

    I., and Xu, H

    Wu, J., Zhang, Z., Feng, Z., Wang, Z., Yang, Z., Jordan, M. I., and Xu, H. Sequential information design: Markov persuasion process and its efficient reinforcement learn- ing.arXiv preprint arXiv:2202.10678,

    arXiv

  6. [27]

    Strategic decision-making in the presence of information asymmetry: Provably ef- ficient rl with algorithmic instruments.arXiv preprint arXiv:2208.11040,

    Yu, M., Yang, Z., and Fan, J. Strategic decision-making in the presence of information asymmetry: Provably ef- ficient rl with algorithmic instruments.arXiv preprint arXiv:2208.11040,

    arXiv

  7. [28]

    Zhong, H., Yang, Z., Wang, Z., and Jordan, M. I. Can reinforcement learning find stackelberg-nash equilibria in general-sum markov games with myopic followers? arXiv preprint arXiv:2112.13521,

    arXiv

  8. [29]

    A posterior sampling frame- work for interactive decision making.arXiv preprint arXiv:2211.01962,

    Zhong, H., Xiong, W., Zheng, S., Wang, L., Wang, Z., Yang, Z., and Zhang, T. A posterior sampling frame- work for interactive decision making.arXiv preprint arXiv:2211.01962,

    arXiv

  9. [30]

    Notation Table For the convenience of the reader, we summarize the notations in the paper as a notation table

    13 The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportability A. Notation Table For the convenience of the reader, we summarize the notations in the paper as a notation table. Notation Explanation S,A, P, R, Hparameters ...

    2018

  10. [31]

    and general function approximation (Russo & Van Roy, 2013; Jiang et al., 2017; Sun et al., 2019; Wang et al., 2020; Jin et al., 2021; Du et al., 2021; Foster et al., 14 The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transport...

    2013

  11. [33]

    by leveraging the techniques in casual inference (Pearl, 2009). There are also a number of works studying the policy optimization in offline RL with a confounded dataset (Wang et al., 2021; Liao et al., 2021; Kallus & Zhou, 2021; Lu et al., 2022; Yu et al., 2022; Wang et al., ...

    2022

  12. [35]

    Section 4.1) and the dynamical transition class (see Section D.1)

    We provide OPME for two types of transition classes: the general transition class (cf. Section 4.1) and the dynamical transition class (see Section D.1). The algorithm OPME-General (OPME-G) is the complete version of the model-based algorithm described in Section 4.1. 17 The S...

    2022

  13. [36]

    that learns the optimal policy of M∗(P t) under the dynamical system transition class. Similar to the reward function, the identification of the transition function can also be established by the NPIV method, that is EM∗(P s) [sh+1 −G ∗ h(sh, ah, eh)|s h, ah] =0.(D.3) For i∈[d...

    2021

  14. [37]

    Denote the least-square estimator of f ∗ by ˆf, standard analysis allows us to bound the projected Mean Squared Error (pMSE) EW EX,Y h Y− ˆf(X)|W i2

    E[Y−f ∗(X)|W] = 0, and construct an empirical dataset for X, Y, Wto perform least-square regression according to the conditional moment equation. Denote the least-square estimator of f ∗ by ˆf, standard analysis allows us to bound the projected Mean Squared Error (pMSE) EW EX,...

    2014

  15. [1982]

    and Dai, B

    Nachum, O. and Dai, B. Reinforcement learning via fenchel- rockafellar duality.arXiv preprint arXiv:2001.01866,

    2001 arXiv

  16. [1984]

    Model-free representation learning and exploration in low-rank mdps.arXiv preprint arXiv:2102.07035,

    Modi, A., Chen, J., Krishnamurthy, A., Jiang, N., and Agarwal, A. Model-free representation learning and exploration in low-rank mdps.arXiv preprint arXiv:2102.07035,

    arXiv

  17. [1994]

    When is partially observable reinforcement learning not scary? arXiv preprint arXiv:2204.08967, 2022a

    Liu, Q., Chung, A., Szepesv ´ari, C., and Jin, C. When is partially observable reinforcement learning not scary? arXiv preprint arXiv:2204.08967, 2022a. Liu, Q., Netrapalli, P., Szepesvari, C., and Jin, C. Opti- mistic mle–a generic model-based algorithm for partially observab...

    arXiv

  18. [1998]

    and Urner, R

    Ben-David, S. and Urner, R. On the hardness of domain adaptation and the utility of unlabeled target samples. In Algorithmic Learning Theory: 23rd International Con- ference, ALT 2012, Lyon, France, October 29-31,

    2012

  19. [2003]

    RL with confounders.Our work is also related to RL with confounders, as the private type th acts as an unobserved confounder in the strategic interaction model

    and optimism in the face of uncertainty principle (Abbasi-Yadkori et al., 2011). RL with confounders.Our work is also related to RL with confounders, as the private type th acts as an unobserved confounder in the strategic interaction model. The off-policy evaluation (OPE) pro...

    2011

  20. [2005]

    A minimax learn- ing approach to off-policy evaluation in partially ob- servable markov decision processes.arXiv preprint arXiv:2111.06784,

    Shi, C., Uehara, M., and Jiang, N. A minimax learn- ing approach to off-policy evaluation in partially ob- servable markov decision processes.arXiv preprint arXiv:2111.06784,

    arXiv

  21. [2007]

    Brero, G., Eden, A., Chakrabarti, D., Gerstgrasser, M., Li, V ., and Parkes, D. C. Learning stackelberg equilibria and applications to economic design games.arXiv preprint arXiv:2210.03852,

    arXiv

  22. [2011]

    F., and Wang, L

    Chen, X., Hu, J., Yang, L. F., and Wang, L. Near-optimal reward-free exploration for linear mixture mdps with plug-in solver.arXiv preprint arXiv:2110.03244,

    arXiv

  23. [2012]

    and Qi, Z

    Chen, X. and Qi, Z. On well-posedness and minimax op- timal rates of nonparametric q-function estimation in off-policy evaluation.arXiv preprint arXiv:2201.06169,

    arXiv

  24. [2014]

    and Kallus, N

    Bennett, A. and Kallus, N. Proximal reinforcement learning: Efficient off-policy evaluation in partially observed markov decision processes.arXiv preprint arXiv:2110.15332,

    arXiv

  25. [2018]

    from Theorem H.7. That, the suboptimality of a uniform policy from {π1, π2, ..., πK} for a given number of episodesKis at most ˜O   PH h=1 HB q dM,hτhC f h log(|R||P||F |/δ) √ K   for OPME-D, and ˜O   PH h=1 B q dV,hτhC f h log(|R||P||G||F |/δ) √ K   for OPME-G by Theo...

    2011

  26. [2019]

    The total variation distance between high-dimensional gaussians

    Devroye, L., Mehrabian, A., and Reddad, T. The total variation distance between high-dimensional gaussians. arXiv preprint arXiv:1810.08693,

    arXiv

  27. [2020]

    and Zhang, B

    Chen, S. and Zhang, B. Estimating and improving dynamic treatment regimes with a time-varying instrumental vari- able.arXiv preprint arXiv:2104.07822,

    arXiv

  28. [2021]

    Off-policy evaluation in infinite-horizon reinforcement learning with latent confounders

    Bennett, A., Kallus, N., Li, L., and Mousavi, A. Off-policy evaluation in infinite-horizon reinforcement learning with latent confounders. InInternational Conference on Artifi- cial Intelligence and Statistics, pp. 1999–2007. PMLR,

    1999

  29. [2022]

    Per- formative reinforcement learning.arXiv preprint arXiv:2207.00046,

    Mandal, D., Triantafyllou, S., and Radanovic, G. Per- formative reinforcement learning.arXiv preprint arXiv:2207.00046,

    arXiv

  30. [2023]

    Instrumental variable value iteration for causal offline reinforcement learning.arXiv preprint arXiv:2102.09907,

    Liao, L., Fu, Z., Yang, Z., Wang, Y ., Kolar, M., and Wang, Z. Instrumental variable value iteration for causal offline reinforcement learning.arXiv preprint arXiv:2102.09907,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.