Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

This paper derives a generalization certificate for RL policies that remains non-vacuous by accounting for Markov dependence through the chain's mixing time, and turns it into a training signal.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 10:17 UTC pith:DMQ7ZH3I

load-bearing objection Genuinely new bound with a fixable proof gap and a real theory-practice mismatch in what the plotted certificates claim; worth a serious referee. the 4 major comments →

arxiv 2510.10544 v3 pith:DMQ7ZH3I submitted 2025-10-12 cs.LG cs.AIstat.ML

PAC-Bayesian Reinforcement Learning Trains Generalizable Policies

Abdelkrim Zitouni , Mehdi Hennequin , Juba Agoun , Ryan Horache , Nadia Kabachi , Omar Rivasplata This is my paper
classification cs.LG cs.AIstat.ML
keywords PAC-Bayesreinforcement learninggeneralization boundsMarkov chainsmixing timenon-vacuous certificatesSoft Actor-Criticcontinuous control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to close a gap in reinforcement learning theory: it wants a rigorous, high-confidence guarantee that a policy trained on a finite set of trajectories will perform well on unseen rollouts, even though RL data are temporally dependent. It derives a PAC-Bayesian generalization bound in which the uncertainty penalty depends explicitly on the mixing time of the Markov chain induced by the policy, so the certificate tightens when the environment mixes quickly. The paper then argues that this bound is non-vacuous, not just in principle but for modern off-policy algorithms like Soft Actor-Critic, and uses it to build PB-SAC, an algorithm that treats the bound as a live objective for exploration and posterior updates. If right, this gives practitioners a computable confidence interval on policy value and converts generalization guarantees from a post-hoc analysis tool into an active part of learning.

Core claim

The central claim is a high-probability bound, stated as Theorem 3.2: with probability at least 1−δ over a dataset of T trajectories of horizon H, the gap between true and empirical expected loss, averaged over a posterior over policies, is at most sqrt{ Rmax^2 τmin (1−γ^{2H}) / (2T(1−γ^2)) · (KL(ρ||μ) + ln(2/δ)) }. The key novelty is that the dependence on the discount factor enters through the finite-horizon factor (1−γ^{2H})/(1−γ^2) rather than the much worse (1−γ)^{-4} style scaling of earlier PAC-Bayes bounds for RL. From this, the paper derives a lower confidence bound on expected return, and claims this certificate is non-vacuous in practice. It then introduces PB-SAC, which maintains

What carries the argument

The engine is the bounded-differences coefficient vector for the negative empirical discounted return: perturbing a single transition at time h of one trajectory changes the averaged return by at most γ^{h-1}R_max/T. Squaring and summing these coefficients gives ||c||^2 = Rmax^2 (1−γ^{2H})/(T(1−γ^2)). This explicit constant plugs into a McDiarmid-type concentration inequality for Markov chains whose dominant parameter is the chain's mixing time τmin — the number of steps after which the chain's state distribution is close to stationary regardless of where it started. The proof feeds that concentration inequality through the standard PAC-Bayesian change-of-measure step, yielding the bound's s

Load-bearing premise

The certificate is only as valid as the algorithm's estimate of the policy-induced chain's mixing time and its guarantee that the prior is not influenced by the training data; the paper's implementation estimates the former from reward autocorrelation and updates the latter with a moving average, so an underestimated τmin or an uncounted data-dependent prior would silently break the high-probability claim.

What would settle it

Run PB-SAC on a small MDP whose transition kernel is known exactly, so the true expected return of any policy can be computed numerically. Repeat the training and certificate computation many times with the algorithm's mixing-time estimate and moving-average prior. If the true expected return falls below the certified lower bound in more than a δ fraction of runs, the theorem, the estimation method, or the prior update is unsound; conversely, if violations stay within δ, the certificate holds.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Rearranged, the theorem gives a lower confidence bound on the expected return of the posterior policy: E_{θ∼ρ} V^{πθ} ≥ empirical discounted return − uncertainty term, so a practitioner can report a certified floor on performance at confidence 1−δ.
  • The uncertainty term scales like sqrt(τmin / T) and sqrt(KL(ρ||μ) / T); more trajectories, shorter horizons, faster mixing, or a prior closer to the posterior all tighten the certificate.
  • Because the bound can be optimized over ρ, the paper's PB-SAC algorithm treats the certificate as a training signal, making exploration uncertainty-aware and balancing empirical return against KL to the prior — with experiments showing match or better sample efficiency than the base actor-critic across continuous-control benchmarks.
  • The paper finds that the certificates remain non-vacuous on HalfCheetah, Hopper, Ant, and Walker2d, and that the gap between empirical and certified return tracks qualitative behavior such as widening during high-variance performance dips and tightening during stable improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The bound's explicit dependence on τmin suggests a new design handle for control: an agent that learns state representations or action choices that mix faster should, for the same dataset, be able to issue tighter certificates. The paper does not test this, but it is a direct consequence of the formula.
  • PB-SAC's moving-average prior is data-dependent, whereas Theorem 3.2 assumes a prior fixed before data are seen. The paper asserts validity without a proof; a strict treatment would need a union bound over the update schedule or a data-dependent-prior argument, and this is the most vulnerable point for someone trying to use the bound as a hard guarantee.
  • The certificate's one-sided robustness to mixing-time error — overestimation loosens but preserves validity, underestimation breaks it — suggests a practical protocol of conservatively overestimating τmin or cross-validating several autocorrelation-based estimates. The paper mentions this but could push it further.
  • A natural next experiment is to apply PB-SAC to sparse-reward tasks, where the posterior-guided exploration might show larger benefits than in the dense-reward MuJoCo suites reported here.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper derives a PAC-Bayesian generalization bound for reinforcement learning that uses Paulin's McDiarmid-type concentration inequality for Markov chains, with an explicit dependence on the chain's mixing time τmin. The bound is intended to hold with probability 1−δ over T collected trajectories of horizon H, and bounds the expected loss gap E_{θ∼ρ}[L(θ)−\hat L_D(θ)] in terms of a KL divergence and a mixing-time-dependent uncertainty term. The authors then introduce PB-SAC, a Soft Actor-Critic variant that periodically computes this bound as a 'certificate' and uses it to guide exploration. Experiments on MuJoCo continuous-control tasks report non-vacuous certificates and competitive performance relative to SAC and PBAC.

Significance. If Theorem 3.2 and its application in PB-SAC were both fully justified, the paper would make a useful contribution: an RL generalization bound with explicit mixing-time dependence and a practical algorithm that produces non-vacuous certificates. The bounded-differences computation for returns (Appendix B.4) is reasonable, and the idea of using Markov-chain concentration within PAC-Bayes is well motivated. However, the paper's headline empirical claim—that PB-SAC produces valid 1−δ confidence certificates—rests on two unproven bridges: a proof gap in the theorem's high-probability conversion, and an algorithmic protocol that violates the theorem's hypotheses (data-dependent prior, estimated rather than known mixing time, off-policy data). These issues are load-bearing, because the reported non-vacuity is exactly what the figures claim.

major comments (4)
  1. [Appendix B.5.4, Eq. (22)→(23)] The proof of Theorem 3.2 jumps from the expectation bound (22) to the high-probability bound (23) by invoking Markov's inequality (Lemma A.1). Markov's inequality applied directly to X = E_{θ∼ρ}[\hat L_D(θ)−L(θ)] gives P(X ≥ u) ≤ E[X]/u, with no logarithmic term. To obtain ln(2/δ), one must apply Markov to an exponential variable, e.g. Y = E_{θ∼μ} exp(κ(\hat L_D(θ)−L(θ))), and use the MGF bound (15). As written, the derivation in B.5.4 is invalid; the theorem may still be correct, but the proof needs a substantial correction.
  2. [Section 4 / Algorithm 1, lines 40–43] Theorem 3.2 explicitly requires the prior µ to be selected independently of data. The paper's own Section 4 states that the prior 'undergoes periodic moving average updates toward the current posterior' to 'prevent KL divergence explosion while preserving bound validity.' Algorithm 1 lines 40–43 implement this by µ←ι·ρ+(1−ι)·µ. No conditioning on the prior, data-splitting argument, or union bound is provided. Since the posterior ρ is itself trained on the replay buffer, the KL(ρ||µ) used in the certificate is data-dependent on both sides, and Theorem 3.2 does not cover this. Consequently, the 'certified discounted return' curves in Figures 1(b) and 4(b) are not established as valid 1−δ lower confidence bounds—yet this is the paper's core empirical claim.
  3. [Section 3.3 / Algorithm 1, line 28] The theorem requires τmin to be a known upper bound on the mixing time of the policy-induced Markov chain for every policy θ in the support of the posterior. The implementation estimates τmin from the autocorrelation of the reward signal of the behavior policy used to collect rollouts. An autocorrelation estimate is not a certified upper bound for all θ ∼ ρ, and underestimation makes the uncertainty term smaller, producing overconfident certificates. The text acknowledges that underestimation 'can be problematic' but only suggests cross-validation; the actual algorithm has no mechanism that turns the estimate into a conservative bound. Thus the plotted certificates incorporate an uncontrolled source of optimism.
  4. [Section 3.1 / Algorithm 1, line 31] Lemma B.1 and Theorem 3.2 are derived for trajectories sampled under the policy π_θ whose return is being evaluated. In PB-SAC, the bound is computed on rollouts collected under a possibly different behavior policy, with importance sampling applied to the returns (Algorithm 1, lines 27–31). The theorem does not cover this off-policy estimator, and the paper provides no concentration guarantee for the importance-sampled empirical return. This is another mismatch between the theory and the object used to produce the reported certificates.
minor comments (5)
  1. [Lemma 3.1, Eq. (6)] The notation H in the sum over h′∈[H] is inconsistent with the earlier use of H as the horizon; the index set is clear but the notation should be unified.
  2. [Section 3.2, Eq. (7)] The definition of c(h,t) uses t for trajectory index whereas the text uses j; align subscripts to avoid confusion.
  3. [Section 4, second paragraph] The phrase 'preserving bound validity' is an assertion, not an argument; either cite a theorem or remove the phrase until a proof is provided.
  4. [Appendix B.2] The step '|G(ξ)−G(ξ̄)| = |γ^{h−1}(R_h−R̄_h)+effects on future rewards|' is imprecise: the future-rewards term is not derived explicitly in the main text, only in B.4. Consider moving B.4's full accounting into B.2.
  5. [General] There are typographical issues, e.g., 'defered' for 'deferred' in Section 3.1, and the sentence 'where The true expected value...' in Appendix B.5.6 starts with a capital 'The' mid-sentence.

Circularity Check

2 steps flagged

PB-SAC's non-vacuous certificates are partly manufactured: the prior is updated toward the posterior and the mixing time is estimated from data, while Theorem 3.2 requires an independent prior and a fixed known τmin.

specific steps
  1. self definitional [Section 4, first paragraph; Algorithm 1, lines 40–43; Theorem 3.2 setup in Section 3.1]
    "The prior µ undergoes periodic moving average updates toward the current posterior with linear decay, preventing KL divergence explosion while preserving bound validity and maintaining exploration capability as the prior stabilizes during training. (Algorithm 1: µ ← ι·ρ + (1−ι)·µ)"

    Theorem 3.2 is derived under 'a prior distribution µ selected independently of data' (Section 3.1). PB-SAC instead defines µ as an exponential moving average of the posterior ρ, which is itself trained on the algorithm's trajectories. The KL(ρ∥µ) term in Eq. (8) is therefore made small by construction, and the tightness of the reported certified lower bound is an artifact of redefining the prior from the posterior. The paper asserts 'preserving bound validity' but provides no data-splitting or data-dependent-prior theorem that would restore the 1−δ guarantee for this moving-average prior.

  2. fitted input called prediction [Section 3.3 'Practical Tractability and Robustness'; Algorithm 1, line 28; Theorem 3.2 and Eq. (32)]
    "We estimate mixing time using autocorrelation decay of the reward signal… However, underestimation can be problematic as it leads to overconfidence.… (Algorithm 1: τ min ← estimate mixing time(D_rollouts))"

    Theorem 3.2 treats τmin as a fixed parameter of the policy-induced Markov chain, but PB-SAC estimates it from reward-signal autocorrelation of the rollouts used to compute the bound, then plugs this estimate into Eq. (32). An autocorrelation-based number is not a certified upper bound on the total-variation mixing time for every θ in the posterior; underestimation shrinks the square-root uncertainty term and inflates the certificate. The paper concedes underestimation 'leads to overconfidence,' so the plotted certified discounted return is not a valid 1−δ lower confidence bound: its tightness is a fitted input, not a known constant.

full rationale

The theoretical Theorem 3.2 is a self-contained PAC-Bayes derivation that combines Paulin's Markov-chain McDiarmid inequality with the bounded-differences coefficients; there is no load-bearing self-citation in that derivation. The circularity is at the algorithm level: the two quantities that make PB-SAC's certificates non-vacuous — the KL to the prior and the mixing time τmin — are both manufactured from data. The prior is updated as a moving average of the posterior, and τmin is estimated from reward autocorrelation, while the theorem's hypotheses require an independent prior and a known mixing-time parameter. Because the paper explicitly relies on these fitted quantities to keep the bound tight ('preventing KL divergence explosion', 'underestimation can be problematic'), the empirical non-vacuity claim in Figures 1(b) and 4(b) is partially circular. The score is 5 rather than higher because Theorem 3.2 itself is an independent theoretical result; only its application in PB-SAC reduces to fitted inputs presented as certified predictions.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central bound rests on Paulin's concentration inequality for uniformly mixing Markov chains, bounded rewards, an on-policy data assumption, and a fixed prior. The algorithm additionally assumes τmin can be estimated from data and that the prior can be adaptively updated without breaking the guarantee; neither is proved, so the numerical certificates are not fully justified.

free parameters (3)
  • τmin (mixing time) = Estimated from reward-signal autocorrelation; no numeric values or rigorous upper bound reported
    The bound's uncertainty term is proportional to sqrt(τmin); underestimation gives overconfident certificates. Section 3.3 admits 'underestimation can be problematic' and offers only a heuristic conservative estimate.
  • Rmax (reward bound) = Not specified in experiments
    Theorem 3.2 requires rewards bounded in [0,Rmax], but the MuJoCo experiments never state the Rmax used to compute the plotted certificates, making the values non-reproducible and potentially chosen post hoc.
  • Prior moving-average decay ι = Initial 0.99, linearly decayed
    Algorithm lines 40-43 update μ toward ρ, directly controlling KL(ρ||μ) and hence the numerical certificate; this hand-chosen schedule is not theoretically justified.
axioms (5)
  • standard math Paulin's McDiarmid-type concentration inequality for Markov chains with bounded differences applies to the empirical discounted return
    Used as the core concentration tool in Section 3.2 and Appendix A.1.2; accepted external theorem.
  • domain assumption The policy-induced Markov chain has finite mixing time τmin < ∞
    The theorem assumes τmin < ∞, but continuous state-action MuJoCo chains are not proved to be uniformly mixing.
  • domain assumption Rewards are bounded in [0,Rmax]
    Required for the bounded-differences constants; not verified for MuJoCo reward functions.
  • domain assumption Training trajectories are generated under the policy π_θ used in the loss (on-policy data)
    Equation (5) defines \hat L_D as the average return under policy π_θ; the algorithm applies the bound to off-policy SAC with importance sampling, outside the theorem's stated setup.
  • ad hoc to paper The prior μ can be updated from data without changing the PAC-Bayes guarantee
    Algorithm lines 40-43 update μ from ρ; Theorem 3.2 requires a prior chosen independently of data, and no extra penalty or union bound is provided for the adaptive prior.

pith-pipeline@v1.3.0-alltime-deepseek · 17286 in / 24408 out tokens · 215607 ms · 2026-08-04T10:17:50.658530+00:00 · methodology

0 comments
read the original abstract

We derive a novel PAC-Bayesian generalization bound for reinforcement learning that explicitly accounts for Markov dependencies in the data, through the chain's mixing time. This contributes to overcoming challenges in obtaining generalization guarantees for reinforcement learning, where the sequential nature of data breaks the independence assumptions underlying classical bounds. The new bound provides non-vacuous certificates for modern off-policy algorithms such as Soft Actor-Critic. We demonstrate the practical utility of the bound through PB-SAC, a novel algorithm that optimizes the bound during training to guide exploration. Experiments across several continuous control tasks show that the proposed approach provides meaningful confidence certificates while maintaining competitive performance.

Figures

Figures reproduced from arXiv: 2510.10544 by Abdelkrim Zitouni, Juba Agoun, Mehdi Hennequin, Nadia Kabachi, Omar Rivasplata, Ryan Horache.

Figure 1
Figure 1. Figure 1: (a) Performance comparison between our PB-SAC, its baseline SAC, and PBAC from Tasdighi et al. (2025); (b) PAC-Bayes analysis of PB-SAC across environments. The empirical discounted return (dashed line) corresponds to Eθ∼ρ[−LˆD(θ)], and the certified discounted return (solid line) corresponds to the lower bound on Eθ∼ρ[−L(θ)] provided by Theorem 3.2 (after rear￾ranging the terms). 6 Conclusion We introduce… view at source ↗
Figure 2
Figure 2. Figure 2: A basic four-state MDP Using a value function V ≡ 0 that assigns zero value to all states, we can compute the Bellman errors: δt(A) = r(A) + γ max a E[V (s ′ )|s = A, a] − V (A) = 0 + 0 · V (C) − 0 = 0 (33) δt(B) = r(B) + γ max a E[V (s ′ )|s = B, a] − V (B) = 0 + 0 · V (D) − 0 = 0 (34) Both states A and B yield the same Bellman error δt = 0 at time t. However, the subsequent errors differ: δt+1(C) = r(C) … view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of our algorithm PB-SAC 19 [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: shows detailed performance comparisons on the Walker2d-v5 environment, demonstrating both the algorithm comparisons (panel a) and the PAC-Bayes analysis (panel b). The empirical discounted return tracks closely with our certified lower bound, validating the theoretical guarantees in practice while consistently maintaining competitive performance in terms of rollout episodic return. 0.0 0.2 0.4 0.6 0.8 1.0 … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Entanglement as a Structural Complexity Axis: A PAC-Bayesian View of Generalization in Quantum Policies and Value Functions

    quant-ph 2026-07 conditional novelty 7.0

    Entanglement raises the Fisher effective dimension of parameterized quantum circuits, producing a PAC-Bayes generalization bound that correctly ranks circuits of identical parameter count by their train-test gap.

Reference graph

Works this paper leans on

38 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Generalization bounds for mixing processes via delayed online-to-PAC conversions

    Baptiste Ab\'el\`es, Eugenio Clerico, and Gergely Neu. Generalization bounds for mixing processes via delayed online-to-PAC conversions. In Gautam Kamath and Po-Ling Loh (eds.), Proceedings of The 36th International Conference on Algorithmic Learning Theory, volume 272 of Proceedings of Machine Learning Research, pp.\ 23--40. PMLR, 24--27 Feb 2025. URL ht...

    2025

  3. [3]

    User-friendly Introduction to PAC-Bayes Bounds

    Pierre Alquier. User-friendly Introduction to PAC-Bayes Bounds . Foundations and Trends® in Machine Learning, 17 0 (2): 0 174–303, 2024. ISSN 1935-8245. doi:10.1561/2200000100. URL http://dx.doi.org/10.1561/2200000100

    doi:10.1561/2200000100 2024

  4. [4]

    Integral Probability Metrics PAC-Bayes Bounds , 2022

    Ron Amit, Baruch Epstein, Shay Moran, and Ron Meir. Integral Probability Metrics PAC-Bayes Bounds , 2022. URL https://arxiv.org/abs/2207.00614

    Pith/arXiv arXiv 2022

  5. [5]

    Weighted sums of certain dependent random variables

    Kazuoki Azuma. Weighted sums of certain dependent random variables. Tohoku Mathematical Journal, 19: 0 357--367, 1967. URL https://api.semanticscholar.org/CorpusID:120707243

    1967

  6. [6]

    PAC-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning

    Olivier Catoni. PAC-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning . IMS Lecture Notes Monograph Series, 56: 0 1–163, 2007. ISSN 0749-2170. doi:10.1214/074921707000000391. URL http://dx.doi.org/10.1214/074921707000000391

    doi:10.1214/074921707000000391 2007

  7. [7]

    M. D. Donsker and S. R. S. Varadhan. Asymptotic evaluation of certain Markov process expectations for large time---I . Communications on Pure and Applied Mathematics, 28 0 (1): 0 1--47, 1975. doi:https://doi.org/10.1002/cpa.3160280102. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/cpa.3160280102

    doi:10.1002/cpa.3160280102 1975

  8. [8]

    M. D. Donsker and S. R. S. Varadhan. Asymptotic evaluation of certain Markov process expectations for large time---IV . Communications on Pure and Applied Mathematics, 36 0 (2): 0 183--212, 1983. doi:https://doi.org/10.1002/cpa.3160360204. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/cpa.3160360204

    doi:10.1002/cpa.3160360204 1983

  9. [9]

    Fard and Joelle Pineau

    M. Fard and Joelle Pineau. PAC-Bayesian Model Selection for Reinforcement Learning . In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010. URL https://proceedings.neurips.cc/paper_files/paper/2010/file/66368270ffd51418ec58bd793f2d9b1b-Paper.pdf

    arXiv 2010

  10. [10]

    PAC-Bayesian Policy Evaluation for Reinforcement Learning , 2012

    Mahdi MIlani Fard, Joelle Pineau, and Csaba Szepesvari. PAC-Bayesian Policy Evaluation for Reinforcement Learning , 2012. URL https://arxiv.org/abs/1202.3717

    Pith/arXiv arXiv 2012

  11. [11]

    On tail probabilities for martingales

    David A Freedman. On tail probabilities for martingales. The Annals of Probability, pp.\ 100--118, 1975. URL https://projecteuclid.org/journals/annals-of-probability/volume-3/issue-1/On-Tail-Probabilities-for-Martingales/10.1214/aop/1176996452.full

    arXiv 1975

  12. [12]

    Risk Bounds for the Majority Vote: From a PAC-Bayesian Analysis to a Learning Algorithm

    Pascal Germain, Alexandre Lacasse, Francois Laviolette, Mario March, and Jean-Francis Roy. Risk Bounds for the Majority Vote: From a PAC-Bayesian Analysis to a Learning Algorithm . Journal of Machine Learning Research, 16 0 (26): 0 787--860, 2015. URL http://jmlr.org/papers/v16/germain15a.html

    2015

  13. [13]

    A Primer on PAC-Bayesian Learning , 2019

    Benjamin Guedj. A Primer on PAC-Bayesian Learning , 2019. URL https://arxiv.org/abs/1901.05353

    Pith/arXiv arXiv 2019

  14. [14]

    Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , 2018

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , 2018. URL https://arxiv.org/abs/1801.01290

    Pith/arXiv arXiv 2018

  15. [15]

    Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisation , 2023

    Maxime Haddouche and Benjamin Guedj. Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisation , 2023. URL https://arxiv.org/abs/2304.07048

    Pith/arXiv arXiv 2023

  16. [16]

    Actor-Critic Algorithms

    Vijay Konda and John Tsitsiklis. Actor-Critic Algorithms . In S. Solla, T. Leen, and K. M\" u ller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https://proceedings.neurips.cc/paper_files/paper/1999/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf

    1999

  17. [17]

    Tutorial on Practical Prediction Theory for Classification

    John Langford. Tutorial on Practical Prediction Theory for Classification . Journal of Machine Learning Research, 6 0 (10): 0 273--306, 2005. URL http://jmlr.org/papers/v6/langford05a.html

    2005

  18. [18]

    Some PAC-Bayesian theorems

    David A McAllester. Some PAC-Bayesian theorems. Machine Learning, 37 0 (3): 0 355--363, December 1999. URL https://link.springer.com/article/10.1023/A:1007618624809. Previously published in conference proceedings of COLT'98

    doi:10.1023/a:1007618624809 1999

  19. [19]

    Riedmiller

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing Atari with Deep Reinforcement Learning . CoRR, abs/1312.5602, 2013. URL http://arxiv.org/abs/1312.5602

    Pith/arXiv arXiv 2013

  20. [20]

    PAC-Bayes Bounds with Data Dependent Priors

    Emilio Parrado-Hern \'a ndez, Amiran Ambroladze, John Shawe-Taylor, and Shiliang Sun. PAC-Bayes Bounds with Data Dependent Priors . Journal of Machine Learning Research, 13 0 (112): 0 3507--3531, 2012. URL http://jmlr.org/papers/v13/parrado12a.html

    2012

  21. [21]

    Concentration inequalities for Markov chains by Marton couplings and spectral methods , 2018

    Daniel Paulin. Concentration inequalities for Markov chains by Marton couplings and spectral methods , 2018. URL https://arxiv.org/abs/1212.2015

    Pith/arXiv arXiv 2018

  22. [22]

    Tighter Risk Certificates for Neural Networks

    Mar \' a P \' e rez - Ortiz, Omar Rivasplata, John Shawe - Taylor, and Csaba Szepesv \' a ri. Tighter Risk Certificates for Neural Networks . Journal of Machine Learning Research, 22 0 (227): 0 1--40, 2021. URL http://jmlr.org/papers/v22/20-879.html

    2021

  23. [23]

    Chromatic PAC-Bayes Bounds for Non-IID Data

    Liva Ralaivola, Marie Szafranski, and Guillaume Stempfel. Chromatic PAC-Bayes Bounds for Non-IID Data . In David van Dyk and Max Welling (eds.), Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pp.\ 416--423, Hilton Clearwater Beach Resort, Clearwater Beach...

    2009

  24. [24]

    PAC-Bayesian Generalisation Error Bounds for Gaussian Process Classification

    Matthias Seeger. PAC-Bayesian Generalisation Error Bounds for Gaussian Process Classification . J. Mach. Learn. Res., 3 0 (null): 0 233–269, March 2003. ISSN 1532-4435. doi:10.1162/153244303765208386. URL https://doi.org/10.1162/153244303765208386

    doi:10.1162/153244303765208386 2003

  25. [25]

    PAC-Bayesian Analysis of Contextual Bandits

    Yevgeny Seldin, Peter Auer, John Shawe-taylor, Ronald Ortner, and Fran c ois Laviolette. PAC-Bayesian Analysis of Contextual Bandits . In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper_files...

    2011

  26. [26]

    PAC-Bayes-Bernstein Inequality for Martingales and its Application to Multiarmed Bandits

    Yevgeny Seldin, Nicolò Cesa-Bianchi, Peter Auer, François Laviolette, and John Shawe-Taylor. PAC-Bayes-Bernstein Inequality for Martingales and its Application to Multiarmed Bandits . In Dorota Glowacka, Louis Dorard, and John Shawe-Taylor (eds.), Proceedings of the Workshop on On-line Trading of Exploration and Exploitation 2, volume 26 of Proceedings of...

    2012

  27. [27]

    Improved PAC-Bayesian Bounds for Linear Regression

    Vera Shalaeva, Alireza Fakhrizadeh Esfahani, Pascal Germain, and Mihaly Petreczky. Improved PAC-Bayesian Bounds for Linear Regression . Proceedings of the AAAI Conference on Artificial Intelligence, 34 0 (04): 0 5660--5667, Apr. 2020. doi:10.1609/aaai.v34i04.6020. URL https://ojs.aaai.org/index.php/AAAI/article/view/6020

    work page doi:10.1609/aaai.v34i04.6020 2020

  28. [28]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and K. M\" u ller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https://proceedings.neurips.cc/paper_files/paper/1999/file/464d828b85b0bed...

    1999

  29. [29]

    Deep Exploration with PAC-Bayes , 2025

    Bahareh Tasdighi, Manuel Haussmann, Nicklas Werge, Yi-Shan Wu, and Melih Kandemir. Deep Exploration with PAC-Bayes , 2025. URL https://arxiv.org/abs/2402.03055

    Pith/arXiv arXiv 2025

  30. [30]

    Lillicrap, and Martin A

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. DeepMind Control Suite . CoRR, abs/1801.00690, 2018. URL http://arxiv.org/abs/1801.00690

    Pith/arXiv arXiv 2018

  31. [31]

    A Strongly Quasiconvex PAC-Bayesian Bound

    Niklas Thiemann, Christian Igel, Olivier Wintenberger, and Yevgeny Seldin. A Strongly Quasiconvex PAC-Bayesian Bound . In Steve Hanneke and Lev Reyzin (eds.), Proceedings of the 28th International Conference on Algorithmic Learning Theory, volume 76 of Proceedings of Machine Learning Research, pp.\ 466--492. PMLR, 15--17 Oct 2017. URL https://proceedings....

    2017

  32. [32]

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: A Standard Interface for Reinforcement Learning Environments , 2024. URL https://arxiv.or...

    Pith/arXiv arXiv 2024

  33. [33]

    Learning via Wasserstein-Based High Probability Generalisation Bounds

    Paul Viallard, Maxime Haddouche, Umut Şimsekli, and Benjamin Guedj. Learning via Wasserstein-Based High Probability Generalisation Bounds . In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  34. [34]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8 0 (3–4): 0 229–256, May 1992. ISSN 0885-6125. doi:10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696

    doi:10.1007/bf00992696 1992

  35. [35]

    Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayesian Theory , 2024

    Zhi Zhang, Chris Chow, Yasi Zhang, Yanchao Sun, Haochen Zhang, Eric Hanchen Jiang, Han Liu, Furong Huang, Yuchen Cui, and Oscar Hernan Madrid Padilla. Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayesian Theory , 2024. URL https://arxiv.org/abs/2411.00401

    Pith/arXiv arXiv 2024

  36. [36]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  37. [37]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  38. [38]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...