Pith. sign in

REVIEW 3 major objections 4 minor 69 references

Universal Value-Function Uncertainties

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Universal value-function uncertainties (UVU) claims that a single TD-trained network's squared error against a fixed random target is, in the infinite-width limit, exactly the variance of an ensemble of universal value functions, giving…

desk verdict Genuinely new and honest, but the central equivalence hinges on an unverified positive-definiteness condition. read the letter →

arxiv 2505.21119 v2 pith:Z6AS377B submitted 2025-05-27 cs.LG cs.AIstat.ML

Moritz A. Zanger , Max Weltevrede , Yaniv Oren , Pascal R. Van der Vaart , Caroline Horsch , Wendelin Böhmer , Matthijs T. J. Spaan This is my paper
classification cs.LGcs.AIstat.ML
keywords universalvalue-functionuncertaintiesepistemicuncertaintyvaluefunctionneuraltangentkerneltemporaldifferencelearningdeepensemblesofflinereinforcementrandomnetworkdistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UVU is a single-model method for estimating epistemic uncertainty in value functions: one online network is trained with temporal-difference learning on synthetic rewards produced by a fixed, randomly initialized target network, and the squared prediction error between the two is the uncertainty estimate. The paper's central claim is that this error is not a heuristic novelty signal but a true value uncertainty. In the infinite-width neural tangent kernel limit, the expected squared UVU error equals the variance of an ensemble of universal Q-functions over random initializations (Corollary 1), and with multiple heads the sample mean error is distributionally identical to an ensemble's sample variance (Corollary 2). If the claim holds, UVU delivers deep-ensemble-grade uncertainty with roughly the compute of one Q-network. The paper supports this with closed-form Gaussian laws for TD-trained value functions and with offline multi-task GoToDoor experiments where UVU matches or exceeds large bootstrapped ensembles.

What carries the argument

The central object is the UVU prediction error $\epsilon(x,\vartheta_t,\psi_0)^2=(u(x,\vartheta_t)-g(x,\psi_0))^2$ between an online TD learner and a fixed random target network, with synthetic rewards $r_g^z(s,a,s',a')=g(s,a,z)-\gamma g(s',a',z)$. The argument is carried by the TD/NTK closed form $f(x,\theta_\infty)=f(x,\theta_0)-\Theta_{xX}(\Theta_{XX}-\gamma\Theta_{X'X})^{-1}(f(X,\theta_0)-(\gamma f(X',\theta_0)+r))$, where $\Theta$ is the neural tangent kernel, the deterministic gradient inner product of infinite-width networks. This identity turns random initializations into a Gaussian distribution over converged value functions, so averaging UVU errors over the two independent networks reproduces the ensemble variance formula. The multi-headed architecture with shared hidden layers is what yields the finite-sample chi-squared equivalence.

What would settle it

Compute the smallest eigenvalue of $\Delta=\Theta_{XX}-\gamma\Theta_{X'X}$ for the neural network and GoToDoor dataset used in Section 5; if it is non-positive, the exponential-convergence condition in Appendix A.3 fails and the equality in Corollary 1 is not guaranteed in the evaluated setting. A complementary check is to compare the expected UVU error with the variance of a large universal-Q ensemble in a controlled wide-network experiment: agreement confirms the NTK limit, while disagreement at practical widths marks where the approximation breaks.

Watch

Extended reading notes

Core claim

Under gradient flow in the infinite-width limit, a neural network trained on a semi-gradient TD loss converges to a deterministic affine function of its random initialization. Applying that identity to UVU, the post-convergence prediction error $\tfrac{1}{2}\epsilon(x,\vartheta_\infty,\psi_0)^2$ has expectation equal to the variance $\mathbb{V}_{\theta_0}[Q(x,\theta_\infty)]$ of universal value functions trained on the same data, and both are Gaussian laws expressible through the neural tangent kernel and the NNGP kernel. The finite-sample version says that the mean squared error of $M$ independent UVU heads is distributed like the sample variance of $M+1$ independently trained Q-functions, a scaled $\chi^2(M)$ distribution. On the offline GoToDoor task-rejection benchmark, UVU reaches average returns on par with or above ensembles of 15 and 35 bootstrapped Q-networks, while running as a single multi-headed model.

Load-bearing premise

The derivation collapses if the matrix $\Delta=\Theta_{XX}-\gamma\Theta_{X'X}$ is not positive definite, because then TD gradient flow need not converge exponentially and the closed-form solution used to equate UVU errors with ensemble variance does not exist; the paper states this condition is stronger than the supervised NTK case and does not verify it for the GoToDoor experiments.

Editorial extensions

If this is right

  • A single UVU model can replace large deep ensembles for value-uncertainty-aware decisions, with compute and memory close to one Q-network rather than an ensemble of many.
  • UVU errors are policy-conditional and long-horizon: they capture uncertainty accumulated over future trajectories, not just immediate novelty as in random network distillation.
  • Theorem 1 provides closed-form Gaussian mean and covariance for semi-gradient TD ensembles, opening value-function ensembles to standard probabilistic tools such as confidence intervals and rejection rules.
  • Corollary 2 transfers ensemble-variance estimators and their confidence properties directly to a single multi-headed model, because the finite-sample distributions match exactly.
  • In offline RL settings such as task rejection, UVU can make conservative decisions without an ensemble penalty; experiments show parity or better relative to large bootstrapped ensembles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the equality survives finite widths better than the NTK caveats suggest, UVU errors could themselves serve as a built-in exploration bonus, since they already aggregate future uncertainty without requiring a separate propagation mechanism.
  • The positive-definiteness condition ties UVU's validity to TD stability; monitoring the spectrum of $\Delta=\Theta_{XX}-\gamma\Theta_{X'X}$ during training could serve as a diagnostic for when UVU errors stop being trustworthy.
  • The paper analyzes offline policy evaluation with fixed data and a fixed policy; extending the equivalence to online control, where the dataset and policy evolve, or to target-network updates, is not established and would be the natural next test.
  • Because the equivalence is distributional rather than only in expectation, UVU could support analytically calibrated rejection thresholds without held-out calibration data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces universal value-function uncertainties (UVU), a single-model method that estimates epistemic value uncertainty as the squared prediction error between an online network u, trained with a semi-gradient temporal-difference loss, and a fixed randomly initialized target network g. The synthetic reward is defined as r_g = g(x) - gamma g(x'), so that g is the value function of the induced reward. Using neural tangent kernel theory in the infinite-width limit, the authors derive a closed-form expression for the post-training distribution of TD-trained networks (Theorem 1), prove that the expected squared UVU error equals the variance of an ensemble of universal Q-functions (Corollary 1), and establish a distributional equivalence between a multi-headed UVU estimator and the sample variance of an M+1 ensemble (Corollary 2). The empirical section evaluates UVU on an offline multitask GoToDoor task-rejection setting and reports performance competitive with large ensembles at lower computational cost.

Significance. If the theoretical equivalence holds, UVU provides a principled single-model alternative to deep ensembles for value-function uncertainty, with the first probabilistic closed-form description of semi-gradient TD ensembles in the NTK regime. The derivation of Corollary 1 is linear-algebraically sound conditional on the stated NTK assumptions and contains no fitted constants; the paper is also transparent about its idealizations in Appendix A.3. The empirical results are a plausible demonstration in a practical offline RL domain. However, the central theorem relies on an unverified and potentially fragile positive-definiteness condition, and the paper does not directly test the claimed equivalence to ensemble variance, so the significance is conditional on these points being addressed.

major comments (3)
  1. [§4, Theorem 1; Appendix A.1.1 and A.3] The positive definiteness of Δ = Θ_XX − γΘ_X′X is a load-bearing assumption that is not stated as a hypothesis in Theorem 1 or Corollaries 1–2. Equation (19) shows that the TD residual decays to zero only if the eigenvalues of −Δ lie in the left half-plane, and the bound in Eq. (25) that justifies the constancy of the NTK requires λ_min(Δ) > 0. The authors acknowledge this condition in Appendix A.3 but do not verify it for the GoToDoor datasets, and the theorem statement as written ('converges to a Gaussian') is not correct without it. Please add the condition as an explicit hypothesis in the theorem and report an empirical check (e.g., the eigenvalue spectrum of the empirical NTK on the actual data and architecture), or provide a theoretical argument that the normalization used in Appendix B.1 guarantees it.
  2. [§5] The main empirical claim is that UVU matches ensemble performance, but the paper never directly tests the central theoretical equivalence of Corollary 1. The task-rejection protocol measures downstream returns, and the width ablation (Fig. 4a) tracks returns rather than the squared-error metric. Since the theory depends on several idealizations not present in the experiments (infinite width, full-batch gradient flow, fixed policy, no target network), a direct comparison of UVU's squared errors with ensemble variance on the same transitions would provide the missing link between Corollary 1 and the practical performance.
  3. [§A.2.3, Corollary 2] The distributional equivalence for multi-headed UVU requires the M prediction errors to be i.i.d. after training, but the heads share hidden-layer parameters. Propositions 2 and 3 establish independence of outputs and gradient kernels at initialization, yet Corollary 2 needs an additional argument that the shared parameters do not induce dependence through the coupled gradient flow. In the infinite-width limit this follows from block-diagonality of the joint NTK, but that step is not proved or even stated; please add a short derivation.
minor comments (4)
  1. [§A.3 and §A.2.3] There are typos: 'initialziation' should be 'initialization' in Appendix A.3, and 'Collorary' should be 'Corollary' in Appendix A.2.3.
  2. [Figure 3 caption] The caption says 'From left to right, (1. and 2.): ... (3. and 4.): ...' but the figure shows only two panels; the numbering is confusing.
  3. [Table 1] The numeric entries contain extra spaces (e.g., '5 .50 ± .15') that make the table hard to read; please format the numbers consistently.
  4. [Equation (4)] The notation 'Θ_XT,X' with a comma is inconsistent with the rest of the paper; use 'Θ_XT X' to match the shorthand defined earlier.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the UVU/ensemble equality is derived from TD/NTK dynamics rather than assumed, and the paper's self-citations are not load-bearing.

full rationale

The central result (Corollary 1) is not circular. The synthetic reward rg = g − γg′ is deliberately constructed so that g is a fixed point of the induced Bellman operator, but that fact alone does not imply the equality E[½ε²] = Var[Q]. The proof computes both sides from the same affine post-training map A (Eq. 29) and the NNGP covariance K, obtaining E[½ε(x,ϑ∞,ψ0)²] = (AKAᵀ)xx = Var[Q(x,θ∞)]; this is a derived identity of the linearized TD/NTK dynamics, not an input assumption. No fitted constants are relabeled as predictions, and the finite-head equivalence in Corollary 2 is likewise derived from the Gaussianity and head-independence of the NTK/NNGP limits. The self-citations (Zanger et al. 2024, 2025; Van der Vaart et al. 2025) appear only for baseline mechanisms or related-work context and do not support the theorem. The positive-definiteness of ΘXX − γΘX′X flagged in Appendix A.3 is an unverified assumption that creates a correctness/robustness risk, but it is not a circularity: the claimed result would simply fail to hold if that assumption were false, rather than being true by construction.

Assumptions & free parameters 3 free parameters · 6 assumptions · 1 invented entities

The theoretical equivalence is parameter-free: the free parameters listed are experimental hyperparameters, not fitted constants in the theorem. The main burdens are idealized assumptions: infinite width, positive definiteness of Δ, full-batch gradient flow, static offline data, and no target network. The invented entity is an algorithmic device, not a new physical object.

free parameters (3)
  • Network width and depth = 512 neurons, 3 hidden layers
    Chosen for the DQN architecture in Appendix B. Empirical performance may depend on it, but the theoretical equivalence is width-agnostic.
  • Number of UVU heads M = 512 heads (Table 3)
    Multi-headed averaging approximates ensemble variance; Corollary 2 holds for any M, so this is a practical choice rather than a fitted constant in the theorem.
  • Learning rate = 3e-4
    Grid-searched on the 10x10 GoToDoor environment; affects empirical results but not the theoretical claim.
assumptions (6)
  • standard math Infinite-width NTK limit with NTK parametrization: the tangent kernel is deterministic and constant throughout training.
    Invoked in Proposition 1 and Theorem 1 via Jacot et al. 2018 and Lee et al. 2020; the paper extends the constancy argument to TD by bounding the training direction.
  • domain assumption The matrix ΘXX − γΘX′X is positive definite so that TD gradient flow converges exponentially.
    Appendix A.1.1 and A.3; the authors acknowledge this is harder to guarantee than in supervised NTK settings and it is not verified empirically.
  • domain assumption Training uses full-batch gradient flow with infinitesimal step size and infinite time.
    Proposition 1 and Theorem 1; the experiments use Adam and minibatches, a disclosed deviation.
  • domain assumption The TD loss uses a semi-gradient stop-gradient operation, not a separate slowly-updated target network.
    Section 2.2 and Appendix A.3; the experiments replace this with a target network in Appendix B.1.
  • domain assumption The dataset is a fixed static offline dataset and the evaluated policy is constant.
    Appendix A.3; online control with changing policies is outside the theorem.
  • domain assumption Online network u, target g, and Q are architecturally equal with i.i.d. Gaussian initialization.
    Corollary 1; needed for the covariance algebra and for independent ensemble heads.
invented entities (1)
  • Fixed randomly initialized target network g acting as a random universal value function
    purpose: Generates synthetic rewards r_g = g(s,a,z) − γg(s',a',z) so that TD residuals of the online learner measure policy-conditional value uncertainty
    This is an algorithmic construction extending RND to the value setting; it exists only inside the paper, though its NTK behavior is characterized analytically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Universal Value-Function Uncertainties." pith.science (2026). https://pith.science/paper/Z6AS377B

@misc{pith2026250521119,
  author       = {Pith},
  title        = {Pith review of: Universal Value-Function Uncertainties},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z6AS377B}},
  note         = {Machine review of arXiv:2505.21119}
}
read the original abstract

Estimating epistemic uncertainty in value functions is a crucial challenge for many aspects of reinforcement learning (RL), including efficient exploration, safe decision-making, and offline RL. While deep ensembles provide a robust method for quantifying value uncertainty, they come with significant computational overhead. Single-model methods, while computationally favorable, often rely on heuristics and typically require additional propagation mechanisms for myopic uncertainty estimates. In this work we introduce universal value-function uncertainties (UVU), which, similar in spirit to random network distillation (RND), quantify uncertainty as squared prediction errors between an online learner and a fixed, randomly initialized target network. Unlike RND, UVU errors reflect policy-conditional value uncertainty, incorporating the future uncertainties any given policy may encounter. This is due to the training procedure employed in UVU: the online network is trained using temporal difference learning with a synthetic reward derived from the fixed, randomly initialized target network. We provide an extensive theoretical analysis of our approach using neural tangent kernel (NTK) theory and show that in the limit of infinite network width, UVU errors are exactly equivalent to the variance of an ensemble of independent universal value functions. Empirically, we show that UVU achieves equal performance to large ensembles on challenging multi-task offline RL settings, while offering simplicity and substantial computational savings.

Figures

Figures reproduced from arXiv: 2505.21119 by the authors.

Figure 2
Figure 2. (left:) Illustration of uncertainty estimation in tabular UVU with 4 independently initialized tables for u and g. Access to full trajectory data allows u to recover g. (right:) By executing action “b”, trajectories are effectively truncated, preventing u from recovering g. All plots use γ = 0.7. exactly, nullifying prediction errors. However, when data coverage is incomplete for the evaluated policy, minimization o… view at source ↗
Figure 1
Figure 1. Chain MDP of length N with unexplored actions b. To build intuition for how UVU operates and captures value uncertainty, we first consider a tabular setting with a simple chain MDP as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. From left to right, (1. and 2.): Variance of an ensemble of 128 universal Q-functions trained on a chain MDP dataset. (3. and 4.): Value uncertainty as measured by UVU prediction errors with a single 128-headed model. All plots evaluate the “a” action of the chain MDP. {(si , ai , zi)} ND i=1 to X ′ = {(s ′ i , a′ i , zi)} ND i=1, where a ′ i ∼ π(·|s ′ i , zi), and rewards r = {ri} ND i=1. The evolution of the param… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: (a) Ablation on GoToDoor-10 with different network widths. Shaded region indicates standard deviations over 5 seeds. (b) Runtime of various ensemble sizes vs. UVU. Ensembles are implemented with vmap in JAX[Bradbury et al., 2018]. (c) Illustration of the GoToDoor envir…
Figure 5
Figure 5. Figure 5: Illustration of the used architecture. ⊙ indicates elementwise multiplication. Data Collection. Our offline datasets are recorded replay buffers from a DQN-agent deployed to the GoToDoor environment with an ϵ-greedy exploration strategy and a particular policy: When th…
Figure 6
Figure 6. Figure 6: Top Row: RND errors. 2nd Row: Value uncertainty as measured by an intrinsic Q-function. 3rd Row: Value uncertainty as measured by an intrinsic Q-function with intrinsic reward priors. and is never evaluated for OOD-actions. To generate reliable uncertainty estimates, R…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 59 canonical work pages

  1. [1]

    G. An, S. Moon, J.-H. Kim, and H. O. Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34: 0 7436--7447, 2021

    work page 2021

  2. [2]

    J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  3. [3]

    Bellemare, S

    M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016

    work page 2016

  4. [4]

    R. Bellman. A Markovian decision process . Journal of mathematics and mechanics, 6, 1957

    work page 1957

  5. [5]

    Bradbury, R

    J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. Vander P las, S. Wanderman- M ilne, and Q. Zhang. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/jax-ml/jax

    work page 2018

  6. [6]

    Burda, H

    Y. Burda, H. Edwards, A. J. Storkey, and O. Klimov. Exploration by random network distillation. In International conference on learning representations, ICLR , 2019

    work page 2019

  7. [7]

    Q. Cai, Z. Yang, J. D. Lee, and Z. Wang. Neural temporal-difference learning converges to global optima. Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  8. [8]

    R. Y. Chen, S. Sidor, P. Abbeel, and J. Schulman. UCB exploration via Q -ensembles. arXiv preprint arXiv:1706.01502, 2017

    arXiv 2017

Show all 69 references
  1. [9]

    X. Chen, C. Wang, Z. Zhou, and K. Ross. Randomized ensembled double Q -learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021

    2021 arXiv

  2. [10]

    Chevalier - Boisvert, B

    M. Chevalier - Boisvert, B. Dai, M. Towers, R. Perez - Vicente, L. Willems, S. Lahlou, S. Pal, P. S. Castro, and J. Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Sys...

    2023

  3. [11]

    D'Angelo and V

    F. D'Angelo and V. Fortuin. Repulsive deep ensembles are bayesian. Advances in Neural Information Processing Systems, 34: 0 3451--3465, 2021

    2021

  4. [12]

    Dearden, N

    R. Dearden, N. Friedman, S. Russell, et al. Bayesian Q -learning. Aaai/iaai, 1998: 0 761--768, 1998

    1998

  5. [13]

    T. G. Dietterich. Ensemble methods in machine learning. In Multiple classifier systems: First international workshop, MCS. Springer, 2000

    2000

  6. [14]

    Engel, S

    Y. Engel, S. Mannor, and R. Meir. Reinforcement learning with gaussian processes. In Proceedings of the 22nd international conference on Machine learning, pages 201--208, 2005

    2005

  7. [15]

    Fellows, K

    M. Fellows, K. Hartikainen, and S. Whiteson. Bayesian Bellman operators. Advances in neural information processing systems, 34, 2021

    2021

  8. [16]

    Fujimoto, W.-D

    S. Fujimoto, W.-D. Chang, E. Smith, S. S. Gu, D. Precup, and D. Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in neural information processing systems, 36: 0 61573--61624, 2023

    2023

  9. [17]

    Gallici, M

    M. Gallici, M. Fellows, B. Ellis, B. Pou, I. Masmitja, J. N. Foerster, and M. Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811, 2024

    2024 arXiv

  10. [18]

    Gerschgorin

    S. Gerschgorin. Uber die abgrenzung der eigenwerte einer matrix. Izvestija Akademii Nauk SSSR, Serija Matematika, 7 0 (3): 0 749--754, 1931

    1931

  11. [19]

    Ghavamzadeh, S

    M. Ghavamzadeh, S. Mannor, J. Pineau, and A. Tamar. Bayesian reinforcement learning: A survey. Foundations and trends in machine learning, 8, 2015

    2015

  12. [20]

    Z. Guo, S. Thakoor, M. P \^ slar, B. Avila Pires, F. Altch \'e , C. Tallec, A. Saade, D. Calandriello, J.-B. Grill, Y. Tang, et al. BYOL-Explore : Exploration by bootstrapped prediction. Advances in neural information processing systems, 35: 0 31855--31870, 2022

    2022

  13. [21]

    H. Hasselt. Double Q -learning. Advances in neural information processing systems, 23, 2010

    2010

  14. [22]

    B. He, B. Lakshminarayanan, and Y. W. Teh. Bayesian deep ensembles via the neural tangent kernel. Advances in neural information processing systems, 33, 2020

    2020

  15. [23]

    K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on Imagenet classification. In Proceedings of the IEEE international conference on computer vision, 2015

    2015

  16. [24]

    C.-J. Hoel, K. Wolff, and L. Laine. Ensemble quantile networks: Uncertainty-aware reinforcement learning with applications in autonomous driving. IEEE Transactions on intelligent transportation systems, 2023

    2023

  17. [25]

    Ishfaq, Q

    H. Ishfaq, Q. Cui, V. Nguyen, A. Ayoub, Z. Yang, Z. Wang, D. Precup, and L. Yang. Randomized exploration in reinforcement learning with general value function approximation. In International conference on machine learning, pages 4607--4616. PMLR, 2021

    2021

  18. [26]

    Ishfaq, Q

    H. Ishfaq, Q. Lan, P. Xu, A. R. Mahmood, D. Precup, A. Anandkumar, and K. Azizzadenesheli. Provable and practical: efficient exploration in reinforcement learning via langevin monte carlo. In International conference on learning representations, 2024 a

    2024

  19. [27]

    Ishfaq, Y

    H. Ishfaq, Y. Tan, Y. Yang, Q. Lan, J. Lu, A. R. Mahmood, D. Precup, and P. Xu. More efficient randomized exploration for reinforcement learning via approximate sampling. In Reinforcement learning conference, 2024 b

    2024

  20. [28]

    Jacot, F

    A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018

    2018

  21. [29]

    D. Janz, J. Hron, P. Mazur, K. Hofmann, J. M. Hern \'a ndez-Lobato, and S. Tschiatschek. Successor uncertainties: Exploration and uncertainty in temporal difference learning. Advances in neural information processing systems, 32, 2019

    2019

  22. [30]

    C. Jin, Z. Allen-Zhu, S. Bubeck, and M. I. Jordan. Is q-learning provably efficient? Advances in neural information processing systems, 31, 2018

    2018

  23. [31]

    C. Jin, Z. Yang, Z. Wang, and M. I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory, pages 2137--2143. PMLR, 2020

    2020

  24. [32]

    Kumar, A

    A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33: 0 1179--1191, 2020

    2020

  25. [33]

    Lahlou, M

    S. Lahlou, M. Jain, H. Nekoei, V. I. Butoi, P. Bertin, J. Rector-Brooks, M. Korablyov, and Y. Bengio. Deup: Direct epistemic uncertainty prediction. arXiv preprint arXiv:2102.08501, 2021

    2021 arXiv

  26. [34]

    Lakshminarayanan, A

    B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017

    2017

  27. [35]

    J. Lee, J. Sohl-dickstein, J. Pennington, R. Novak, S. Schoenholz, and Y. Bahri. Deep neural networks as gaussian processes. In International conference on learning representations, 2018

    2018

  28. [36]

    J. Lee, L. Xiao, S. S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein , and J. Pennington. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent . Journal of Statistical Mechanics: Theory and Experiment, 2020, Dec. 2020

    2020

  29. [37]

    K. Lee, M. Laskin, A. Srinivas, and P. Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, pages 6131--6141. PMLR, 2021

    2021

  30. [38]

    S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702--1712. PMLR, 2022

    2022

  31. [39]

    C. Liu, L. Zhu, and M. Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. Advances in Neural Information Processing Systems, 33: 0 15954--15964, 2020

    2020

  32. [40]

    C. E. Luis, A. G. Bottero, J. Vinogradska, F. Berkenkamp, and J. Peters. Model-based uncertainty in value functions. In International Conference on Artificial Intelligence and Statistics, pages 8029--8052. PMLR, 2023

    2023

  33. [41]

    L \"u tjens, M

    B. L \"u tjens, M. Everett, and J. P. How. Safe reinforcement learning with model uncertainty estimates. In 2019 International Conference on Robotics and Automation (ICRA), pages 8662--8668. IEEE, 2019

    2019

  34. [42]

    C. Lyle, M. Rowland, W. Dabney, M. Kwiatkowska, and Y. Gal. Learning dynamics and generalization in reinforcement learning. arXiv preprint arXiv:2206.02126, 2022

    2022 arXiv

  35. [43]

    V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518, 2015

    2015

  36. [44]

    M. S. A. Nadeem, J.-D. Zucker, and B. Hanczar. Accuracy-rejection curves (arcs) for comparing classification methods with a reject option. In Machine Learning in Systems Biology, pages 65--81. PMLR, 2009

    2009

  37. [45]

    Nikolov, J

    N. Nikolov, J. Kirschner, F. Berkenkamp, and A. Krause. Information-directed exploration for deep reinforcement learning. In International conference on learning representations, ICLR , 2019

    2019

  38. [46]

    Osband, C

    I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN . Advances in neural information processing systems, 29, 2016

    2016

  39. [47]

    Osband, B

    I. Osband, B. Van Roy, D. J. Russo, Z. Wen, et al. Deep exploration via randomized value functions. Journal of machine learning research, 20, 2019

    2019

  40. [48]

    O’Donoghue, I

    B. O’Donoghue, I. Osband, R. Munos, and V. Mnih. The uncertainty Bellman equation and exploration. In International conference on machine learning. PMLR, 2018

    2018

  41. [49]

    Pathak, P

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning. PMLR , 2017

    2017

  42. [50]

    Rashid, B

    T. Rashid, B. Peng, W. B \"o hmer, and S. Whiteson. Optimistic exploration even with a pessimistic initialisation. Proceedings of ICLR 2020, 2020

    2020

  43. [51]

    D. Russo. Worst-case regret bounds for exploration via randomized value functions. Advances in Neural Information Processing Systems, 32, 2019

    2019

  44. [52]

    Schaul, D

    T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International conference on machine learning, pages 1312--1320. PMLR, 2015

    2015

  45. [53]

    Schmitt, J

    S. Schmitt, J. Shawe-Taylor, and H. van Hasselt. Exploration via epistemic value estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2023

    2023

  46. [54]

    Schwarzer, A

    M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929, 2020

    2007 arXiv

  47. [55]

    Silver, A

    D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529 0 (7587): 0 484--489, 2016

    2016

  48. [56]

    A. L. Strehl, L. Li, E. Wiewiora, J. Langford, and M. L. Littman. Pac model-free reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pages 881--888, 2006

    2006

  49. [57]

    Touati and Y

    A. Touati and Y. Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34: 0 13--23, 2021

    2021

  50. [58]

    Tsilivis and J

    N. Tsilivis and J. Kempe. What can the neural tangent kernel tell us about adversarial robustness? Advances in Neural Information Processing Systems, 35: 0 18116--18130, 2022

    2022

  51. [59]

    P. R. Van der Vaart, M. T. Spaan, and N. Yorke-Smith. Epistemic B ellman operators. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025

    2025

  52. [60]

    Vinyals, I

    O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575 0 (7782): 0 350--354, 2019

    2019

  53. [61]

    H.-T. Wai, Z. Yang, Z. Wang, and M. Hong. Provably efficient neural gtd for off-policy learning. Advances in Neural Information Processing Systems, 33: 0 10431--10442, 2020

    2020

  54. [62]

    C. Xiao, B. Dai, J. Mei, O. A. Ramirez, R. Gummadi, C. Harris, and D. Schuurmans. Understanding and leveraging overparameterization in recursive value estimation. In International Conference on Learning Representations, 2021

    2021

  55. [63]

    G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv:1902.04760, 2019

    1902 arXiv

  56. [64]

    Z. Yang, C. Jin, Z. Wang, M. Wang, and M. Jordan. Provably efficient reinforcement learning with kernel and neural function approximations. Advances in Neural Information Processing Systems, 33: 0 13903--13916, 2020

    2020

  57. [65]

    Y. Yue, R. Lu, B. Kang, S. Song, and G. Huang. Understanding, predicting and better resolving q-value divergence in offline-rl. Advances in Neural Information Processing Systems, 36: 0 60247--60277, 2023

    2023

  58. [66]

    M. A. Zanger, W. B \"o hmer, and M. T. Spaan. Diverse projection ensembles for distributional reinforcement learning. In International conference on learning representations, 2024

    2024

  59. [67]

    M. A. Zanger, P. R. Van der Vaart, W. B \"o hmer, and M. T. Spaan. Contextual similarity distillation: Ensemble uncertainties with a single model. arXiv preprint arXiv:2503.11339, 2025

    2025 arXiv

  60. [68]

    Zheng, R

    C. Zheng, R. Salakhutdinov, and B. Eysenbach. Contrastive difference predictive coding. arXiv preprint arXiv:2310.20141, 2023

    2023

  61. [69]

    Q. Zhou, H. Li, and J. Wang. Deep model-based reinforcement learning via estimated uncertainty and conservative policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.