Pith. sign in

REVIEW 5 major objections 6 minor 67 references

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An adaptive Q-value-and-gradient stop rule makes off-policy deep RL learn faster and finish with higher scores.

desk verdict A useful, clearly-written adaptive early-stopping mechanism with real empirical gains, but the stopping rule is self-referential and the paper does not yet show it beats simple truncation baselines. read the letter →

arxiv 2506.13672 v1 pith:6RBFIKPY submitted 2025-06-16 cs.LG

Jiashun Liu , Johan Obando-Ceron , Pablo Samuel Castro , Aaron Courville , Ling Pan This is my paper
classification cs.LG
keywords deepreinforcementlearningsampleefficiencysunkcostfallacyearlyepisodeterminationadaptivestoppingthresholdreplaybuffercontinuouscontroloff-policy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Off-policy deep reinforcement learning agents normally run every episode to completion, even when the trajectory has become a familiar dead end; the paper calls this the sunk cost fallacy of deep RL and argues it wastes environment interactions and pollutes the replay buffer. The paper's claim is that a lightweight early-termination layer, LEAST, can cure this by stopping and resetting an episode when the current Q-value falls below an adaptive threshold built from the median Q-values of recent episodes and modulated by critic-gradient magnitude. Because the threshold is step-dependent and sensitive to learning potential, LEAST is meant to preserve exploration of genuinely novel states while cutting short repetitions of low-value behavior. The paper reports that wrapping TD3, SAC, REDQ, and DrQv2 with LEAST improves both learning efficiency and final performance on MuJoCo and DeepMind Control Suite tasks, without adding trainable networks. If correct, this identifies enforced episode completion as a removable cost shared across many existing algorithms.

What carries the argument

The load-bearing object is the adaptive stopping threshold defined by Eq. (1) and Eq. (3): stop and reset when the current Q-value $\hat{Q}_i$ falls below a step-dependent threshold $\omega_i \epsilon_i$, with $\epsilon_i$ the median of recent Q-values at that step and $\omega_i$ the ratio of the recent median gradient magnitude to the current gradient magnitude. The median rather than the mean keeps the statistic robust to outlier Q-values; step-dependence matters because early and late episode steps have different Q-scales; and the gradient ratio is a proxy for learning potential, so high-uncertainty states avoid premature truncation. Around that decision rule sit two supporting mechanisms: entropy-based resizing of the recent-episode reflection sets $B_Q$ and $B_G$, and an exploration-noise schedule that raises action noise when early stops become frequent.

What would settle it

Take a task where a good episode must first pass through a low-Q region, run LEAST while logging every stop decision, and compare each truncated episode against an oracle policy's estimate of the return it would have earned if continued; if a large fraction of truncated episodes would have finished above the current median, the below-median stop rule is cutting informative trajectories and the claimed efficiency gain would not hold.

Watch

Extended reading notes

Core claim

The paper argues that fixed-horizon off-policy agents display a sunk cost fallacy: they keep rolling through low-quality trajectories already well represented in the replay buffer, wasting interaction budget and adding uninformative transitions. LEAST stops and resets at step $i$ when $\hat{Q}_i < \omega_i \epsilon_i$ (for $\epsilon_i \ge 0$), where $\epsilon_i = \mathrm{Median}(B_Q[:,i])$ over the $K$ most recent episodes and $\omega_i = \mathrm{Median}(B_G[:,i]) / G_i$ is the ratio of the recent median critic-gradient magnitude to the current one; for negative $\epsilon_i$, the inverse weight is used. The weight prevents stopping in unfamiliar states that still offer learning signal, while a pure below-median Q-value triggers truncation of well-known low-value behavior. The reported result is that this rule, without any added trainable component, raises learning speed and final scores for TD3, SAC, REDQ, and DrQv2 on MuJoCo and visual DeepMind Control tasks.

Load-bearing premise

The whole rule assumes the per-step median of recently seen Q-values is a trustworthy baseline for what counts as unproductive, so a current Q-value below that median justifies stopping; early in training, when Q-values are noisy and the same policy sets the median, a bad policy can make its own bad trajectories look normal.

Editorial extensions

If this is right

  • On the MuJoCo tasks tested, adding LEAST to TD3, SAC, and REDQ speeds early learning and raises final normalized scores, with REDQ plus LEAST reaching the highest overall scores among the compared variants.
  • On image-based DeepMind Control tasks, LEAST added to DrQv2 accelerates convergence, about 30 percent faster on Finger Turn Hard, and outperforms CURL and A-LIX while approaching TACO without adding any trainable module.
  • The replay buffer of a LEAST agent contains a higher share of transitions with high Q-value and strong learning signal, which is the mechanism the paper gives for why fewer environment interactions are wasted.
  • On a long-horizon Humanoid task, LEAST slows the decline of the fraction of active units, which the paper reads as evidence that avoiding uninformative early data helps preserve network plasticity over long training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The median threshold is computed from the very policy being evaluated, so a policy that degrades drags its own reference down; a direct comparison to a threshold anchored on a fixed or target-network policy would show how much of LEAST's gain comes from adaptivity rather than from the absolute threshold level.
  • Because LEAST works on the interaction loop rather than on the network, combining it with buffer-curation ideas such as prioritized sampling, small batches, or dropout critics could compound rather than saturate sample-efficiency gains, but the paper does not test these combinations.
  • The sunk-cost framing points to a direct head-to-head with learned episode-length and reset-based exploration methods; matching those baselines with fewer environment interactions would isolate early stopping as the active ingredient.
  • A sparse-reward environment would stress Eq. (3): if LEAST is correct, the gradient-modulated threshold should keep early Q-values from triggering mass termination, and any failure there would indicate the median baseline needs a policy-independent warm-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes LEAST, a lightweight mechanism for early episode termination in off-policy deep RL. The stopping decision is based on the current Q-value relative to a step-dependent median of recent Q-values, modulated by a gradient-magnitude-based learning-potential weight, plus entropy-based buffer resizing and adaptive exploration noise. The method is evaluated on MuJoCo benchmarks with TD3, SAC, REDQ, and DroQ, and on DeepMind Control Suite image-based tasks with DrQv2, reporting improved learning efficiency and final performance. The authors frame the contribution as overcoming the sunk cost fallacy in deep RL, where agents continue unproductive episodes because they lack autonomous stopping ability.

Significance. If the empirical gains survive proper controls, LEAST is an interesting and orthogonal contribution to sample-efficiency research: it requires no additional networks, is simple to implement, and the paper provides pseudocode, module ablations, and experiments across several off-policy algorithms and benchmark families. The plasticity analysis in Appendix C.1 is a useful addition. However, the central mechanism is under-validated against alternative explanations such as truncation effects, time-limit bias, and the self-referential nature of the threshold, and the claimed statistical significance is not supported by tests. These issues are central, not peripheral, to the paper's main claim.

major comments (5)
  1. [§3.2, Eq. (1), Eq. (3), Algorithm 1] The stopping threshold is self-referential in a way that can extinguish the stop signal exactly when it is needed. When a policy is stuck in a low-reward region, recent episodes also have low Q-values, so the median reference BQ[:, i] drops and the condition Qhat_i < median becomes harder to satisfy; the entropy-based buffer resizing in §3.2 does not break this feedback because it only changes the window size. The paper provides no diagnostic test that stopping decisions actually correlate with low remaining return or with low trajectory quality. Please add a direct measurement (e.g., compare Q-value distributions and achieved future returns for stopped vs. continued trajectories) and include a stress-test scenario where the policy is known to be stuck.
  2. [§4.1, Figures 6-8; §4.2, Figures 9-10] There is no fixed or random truncation control. Early stopping can improve learning even if the stopping rule is uninformative, because truncation changes the effective time horizon, introduces a time-limit bias (Pardo et al., 2018), and acts as a form of implicit regularization. The paper cites Pardo et al. but does not analyze how the induced bias affects the Q-values that are themselves used in the threshold. Please include a baseline that stops at a fixed fraction of the episode or at random steps with the same expected length, and report the average episode length under LEAST for each task.
  3. [Abstract, §4.1, §4.2, Conclusion] The paper repeatedly claims that LEAST 'significantly improves' performance, but no statistical significance tests are reported. Figures 6 and 9 show overlapping standard deviations on several tasks (e.g., HalfCheetah and Quadruped Run), and the normalized-score box plots in Figures 8 and 10 are based on only 5 seeds. Please report per-seed paired differences with bootstrap confidence intervals or a paired test across seeds, and soften the 'significant' language unless such tests are provided.
  4. [Appendix B.2, §4.7, Tables 4-5] The claim of generality is weakened by per-task hyperparameter tuning. Start times differ per algorithm and per task (0.6M for most MuJoCo tasks but 0.75M for Ant and 0.25M for HalfCheetah; 0.5M for SAC/REDQ with exceptions; 0.4M for DMC), the learning-potential weight omega is 0.5 for most tasks but 0.7 for Humanoid and 0.6 for HalfCheetah in SAC, and the initial reflection-set size differs between state-based (150) and image-based (250) tasks. The sentence 'We suggest using seed 0 → 7 to reproduce the learning curve' in Appendix B.2 is particularly concerning, as it suggests the reported curves may depend on a specific seed choice. Please clarify whether reported curves are averages over seeds or representative seeds, and provide per-seed results rather than a suggested seed range.
  5. [§3.2, Eq. (2), Figure 4] The premise that gradient magnitude of the critic is a useful proxy for learning potential is asserted without validation. Eq. (2) uses the ratio of historical median gradient to current gradient, but no experiment tests whether larger gradients actually correspond to more informative transitions or to larger expected improvement in the value estimate. Figure 4 compares module variants but does not isolate this premise. Please add an analysis of the correlation between gradient magnitude and actual TD-error reduction or future return, or an ablation in which omega is replaced by a randomized weight to ensure the performance gain is not just from the quality component or from the clipping operation.
minor comments (6)
  1. [Algorithm 1, line 9; §3.2, Eq. (2)] The notation is inconsistent: Algorithm 1 fills BG with the TD error Li, while §3.2 says BG stores gradient magnitudes and Eq. (2) uses a current gradient magnitude Gi. Please clarify that Li is the gradient norm of the TD loss or rename the variable to match.
  2. [Footnote 1] Footnote 1 says 'Please refer to Appendix 5 for detailed related work,' but the related work is Section 5, not an appendix; please correct the cross-reference.
  3. [§3.1 vs. Appendix A.2] The maze horizon is inconsistent: §3.1 describes PointMaze variants with horizon lengths 10, 18, and 24, while Appendix A.2 says the episode length is 50 steps; please clarify which horizon corresponds to each variant.
  4. [Figure 7] The caption of Figure 7 says the plot reports the average number of training steps required to reach the maximum normalized score achieved by the vanilla variant, but it is unclear whether this is the first time that score is reached or the best score during training; please specify the measurement protocol.
  5. [§4.2] The claim that convergence is 'approximately 30% faster' on Finger Turn Hard is not accompanied by a definition of convergence speed or confidence intervals; please state how the 30% was computed.
  6. [Epigraph] The epigraph reads 'Halr Arkes' but the reference is to Hal R. Arkes; please fix the typo.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the benchmark comparisons are external and the self-referential threshold is a design choice, not a fitted prediction.

full rationale

The paper's central claim is empirical: LEAST uses a hand-specified stopping rule (Eqs. 1 and 3) computed from recent Q-values and gradient magnitudes, and the paper reports that this improves learning efficiency and final performance on MuJoCo and DeepMind Control benchmarks relative to vanilla TD3, SAC, REDQ, and DrQv2. The stopping threshold is endogenous to the agent's own value estimates, which can create a feedback loop if a poor policy lowers the median threshold; however, that is a robustness concern about the heuristic, not a circular derivation. The paper does not fit a parameter to the benchmark outcome and then rename that fit as a prediction, and it does not validate the stopping rule against a definition of "unproductive" that is itself defined by the rule. The Figure 3 observation that low-Q samples are less frequent in the buffer is a near-description of the stopping rule rather than an independent confirmation, but the headline performance result is not entailed by that description and is supported by external comparisons. Self-citations such as Ceron et al. (2023) are motivational background and are not load-bearing for the main claim, and no uniqueness theorem or ansatz is imported from prior author work. The derivation chain therefore does not reduce to its own inputs by construction.

Assumptions & free parameters 9 free parameters · 4 assumptions · 0 invented entities

The method does not introduce new physical or mathematical entities. It relies on existing RL quantities (Q-values, gradients) and several hand-set hyperparameters that are tuned per task, along with domain assumptions about free resets and the validity of Q-values and gradients as quality signals.

free parameters (9)
  • LEAST start time (tstart) = TD3: 0.6M (most), Ant 0.75M, HalfCheetah 0.25M; SAC: 0.5M (most), Ant 0.65M, HalfCheetah 0.25M; DMC: 0.4M
    Controls when LEAST activates during training; tuned per task in Appendix B.2, which affects the results.
  • Initial reflection set size = 150 for MuJoCo, 250 for image-based DMC
    Appendix B.2 and Table 5 show performance varies with this value; authors recommend 250 episodes.
  • Learning potential weight scale (omega) = 0.5 default, 0.7 for Humanoid and 0.6 for HalfCheetah in SAC
    Section 4.4 shows sensitivity; best range [0.3, 0.6], with task-specific values chosen separately.
  • Noise upper bound (sigma_bar) = TD3: [0.25, 0.35]; SAC: [0.15, 0.25]
    Table 1 ablates this parameter; the selected range depends on the algorithm.
  • Stop length threshold (e) and window (m) = e in [400, 500], m = 50
    Table 2 ablates e; m is stated as 50 in Section 4.5.
  • Entropy overflow rate (gamma) = gamma in [0, 0.1] recommended, 0.05 in main runs
    Table 3 ablates gamma; larger values hurt performance.
  • Threshold clipping constants = max = max Q in BQ, min = min Q in BQ; alternative '0.75*median Q' and '0.25*median Q'
    Appendix B.2 describes heuristic clipping limits for omega times epsilon; values are not fixed in the paper.
  • Noise schedule temperatures (tau, mu) = Not reported
    Eq. 4 uses tau and mu as 'pre-set temperatures', but no concrete values are given, making replication harder.
  • Entropy baseline (H_bar) and adjustment scale (h) = Not reported
    Algorithm 1 uses H_bar and h for dynamic buffer resizing, but the paper does not specify their values.
assumptions (4)
  • domain assumption The environment permits free reset at any time during an episode without penalty.
    LEAST stops and resets episodes mid-rollout; this is valid in MuJoCo/DMC but not in all real-world settings. Invoked throughout Section 3.2 and Algorithm 1.
  • domain assumption Learned Q-values are a reliable enough indicator of trajectory quality to serve as a stopping signal.
    Eq. 1 and Eq. 3 compare current Q-hat to historical medians; early-training Q estimates are noisy, which the paper acknowledges in Section 3.2 but does not resolve.
  • ad hoc to paper Gradient magnitude of the critic is a useful proxy for learning potential.
    Section 3.2 justifies larger gradients as indicating novel or poorly understood states, borrowing from loss-based prioritization literature; this is a heuristic without formal backing.
  • domain assumption The median of recent episodes at a given step is a stable and appropriate baseline for comparison.
    Section 4.3 shows median beats mean on one task, but there is no theoretical justification and the median itself is non-stationary during training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/6RBFIKPY

@misc{pith2026250613672,
  author       = {Pith},
  title        = {Pith review of: The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6RBFIKPY}},
  note         = {Machine review of arXiv:2506.13672}
}
read the original abstract

Off-policy deep reinforcement learning (RL) typically leverages replay buffers for reusing past experiences during learning. This can help improve sample efficiency when the collected data is informative and aligned with the learning objectives; when that is not the case, it can have the effect of "polluting" the replay buffer with data which can exacerbate optimization challenges in addition to wasting environment interactions due to wasteful sampling. We argue that sampling these uninformative and wasteful transitions can be avoided by addressing the sunk cost fallacy, which, in the context of deep RL, is the tendency towards continuing an episode until termination. To address this, we propose learn to stop (LEAST), a lightweight mechanism that enables strategic early episode termination based on Q-value and gradient statistics, which helps agents recognize when to terminate unproductive episodes early. We demonstrate that our method improves learning efficiency on a variety of RL algorithms, evaluated on both the MuJoCo and DeepMind Control Suite benchmarks.

Figures

Figures reproduced from arXiv: 2506.13672 by the authors.

Figure 1
Figure 1. LEAST enables the agent to prematurely end the current episode by monitoring the quality of present situations, such as getting stuck in a suboptimal trajectory. This mechanism improves sample efficiency, reduces replay buffer contamination, and con￾serves the overall interaction budget. processes. A simple example of this is watching a bad movie until the end because we have already paid for a movie ticket, thereby… view at source ↗
Figure 2
Figure 2. demonstrates significantly higher learning effi￾ciency of the advanced agent with auto-stopping capability compared to the vanilla agent. As the horizon of the maze expands, interaction budgets become more constrained, lead￾ing to a correspondingly greater demand for sample effi￾ciency. The vanilla agent’s performance is significantly reduced (maintaining at 60 score), which proves that it is trapped in suboptimal t… view at source ↗
Figure 3
Figure 3. Replay buffer data distribution at 2 × 105 timesteps. Compared to the vanilla agent, the advanced agent with LEAST sig￾nificantly reduces the proportion of low-quality transitions (white region: low Q-value and low learning signal) and increases the density of high-quality samples (black region: high Q-value and high gradient magnitude), resulting in a more informative and effi￾cient training dataset. 3.2. Proposed … view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Validation of modules, normalized performance on a high-dim task, i.e., large maze (10 seeds). The box records the final scores of all seeds, the black line inside the box represents the median, while the horizontal line outside the box records the maximum and minimum …
Figure 5
Figure 5. Figure 5: Schedule of exploration noise adjustment. As the fre￾quency of premature stopping increases, indicating potential stag￾nation in policy improvement, the standard deviation of the ex￾ploration noise is gradually increased to encourage behavioral diversification and esca…
Figure 6
Figure 6. Figure 6: Empirical validation of LEAST across diverse deep RL algorithms in MuJoCo environments. Results show mean ± standard deviation over 5 runs. LEAST consistently improves both sample efficiency and final performance across: (a) TD3, representing deterministic policy metho…
Figure 9
Figure 9. Figure 9: Performance on image-based control tasks. The hor￾izontal axis represents training steps; the vertical axis shows the cumulative score. Each curve depicts the mean performance over 5 random seeds, with shaded regions indicating standard deviation 0.55 0.60 0.65 0.70 0.…
Figure 10
Figure 10. Figure 10: Final performance on DMC tasks. Results are ag￾gregated using the same metric as in [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 8
Figure 8. Figure 8: Final performance on MuJoCo task, aggregated using the same evaluation metrics as in [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 11
Figure 11. Figure 11: Comparison of methods for estimating the central tendency of BQ on the Ant task. Using the median provides more stable and robust estimates than the arithmetic mean, particularly in the presence of outliers. This stability helps mitigate overesti￾mation in early stopp…
Figure 12
Figure 12. Figure 12: Sensitivity analysis of the weighting factor ω in the adaptive stopping threshold of LEAST. We vary the scaling coefficient applied to ω across a range of values and evaluate its impact on final performance. Results are averaged over 5 seeds. This analysis reveals tha…
Figure 13
Figure 13. Figure 13: Layout of the maze. The agent, a 2D point, is noisily initialized in the bottom left. At test time, we evaluate it on reaching the right corner. The 2-D state and action space correspond to the planar position and velocity, respectively. The episode length is 50 steps…
Figure 14
Figure 14. Figure 14: Statistics of the final positions of two agents on the medium task. A.4. Sunk Cost Fallacy Sunk cost fallacy (Turpin et al., 2019), also known as sunk cost bias (Arkes & Blumer, 1985), is widely discussed in cognitive science. It describes the tendency to follow throu…
Figure 15
Figure 15. Figure 15: Performance on long-term training task. The curve and shade denote the mean and the standard deviation over 5 runs. C.2. Ablation Study of Update-to-Data We use TD3 as the backbone and try to improve UTD to enable fast iteration of policies to better capture the advan…
Figure 16
Figure 16. Figure 16: Sensitivity to UTD. Average of 5 runs on Ant. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 41 canonical work pages

  1. [1]

    Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, pp.\ 620--636. PMLR, 2023

    work page 2023

  2. [2]

    Arkes, H. R. and Blumer, C. The psychology of sunk cost. Organizational behavior and human decision processes, 35 0 (1): 0 124--140, 1985

    work page 1985

  3. [3]

    Openai gym

    Brockman, G. Openai gym. arXiv preprint arXiv:1606.01540, 2016

    arXiv 2016

  4. [4]

    Ceron, J. S. O. and Castro, P. S. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, pp.\ 1373--1383. PMLR, 2021

    work page 2021

  5. [5]

    Ceron, J. S. O., Bellemare, M. G., and Castro, P. S. Small batch deep reinforcement learning. In NeurIPS, 2023

    work page 2023

  6. [6]

    Ceron, J. S. O., Ara \'u jo, J. G. M., Courville, A., and Castro, P. S. On the consistency of hyper-parameter selection in value-based deep reinforcement learning. In Reinforcement Learning Conference, 2024

    work page 2024

  7. [7]

    J., Roberts, S., and Celiktutan, O

    Cetin, E., Ball, P. J., Roberts, S., and Celiktutan, O. Stabilizing off-policy deep reinforcement learning from pixels. arXiv preprint arXiv:2207.00986, 2022

    arXiv 2022

  8. [8]

    Randomized ensembled double q-learning: Learning fast without a model

    Chen, X., Wang, C., Zhou, Z., and Ross, K. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021

    arXiv 2021

Show all 67 references
  1. [9]

    Model-free generative replay for lifelong reinforcement learning: Application to starcraft-2

    Daniels, Z., Raghavan, A., Hostetler, J., Rahman, A., Sur, I., Piacentino, M., and Divakaran, A. Model-free generative replay for lifelong reinforcement learning: Application to starcraft-2. arXiv preprint arXiv:2208.05056, 2022

    2022 arXiv

  2. [10]

    Dijkstra, K. A. and Hong, Y.-y. The feeling of throwing good money after bad: The role of affective reaction in the sunk-cost fallacy. PLOS ONE, 14 0 (1): 0 1--19, 01 2019. URL https://doi.org/10.1371/journal.pone.0209900

    2019 doi

  3. [11]

    G., and Courville, A

    D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Bellemare, M. G., and Courville, A. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022

    2022

  4. [12]

    W., Subramanian, J., and Ghassemi, M

    Fatemi, M., Killian, T. W., Subramanian, J., and Ghassemi, M. Medical dead-ends and learning to identify high-risk states and treatments. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volu...

    2021

  5. [13]

    Revisiting fundamentals of experience replay

    Fedus, W., Ramachandran, P., Agarwal, R., Bengio, Y., Larochelle, H., Rowland, M., and Dabney, W. Revisiting fundamentals of experience replay. In International conference on machine learning, pp.\ 3061--3071. PMLR, 2020

    2020

  6. [14]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.\ 1587--1596. PMLR, 2018

    2018

  7. [15]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. PMLR, 2018

    2018

  8. [16]

    Sunk-cost fallacy and cognitive ability in individual decision-making

    Haita-Falah, C. Sunk-cost fallacy and cognitive ability in individual decision-making. Journal of Economic Psychology, 58: 0 44--59, 2017. ISSN 0167-4870. doi:https://doi.org/10.1016/j.joep.2016.12.001. URL https://www.sciencedirect.com/science/article/pii/S0167487016307346

    2017 doi

  9. [17]

    R., Millman, K

    Harris, C. R., Millman, K. J., Van Der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. Array programming with numpy. Nature, 585 0 (7825): 0 357--362, 2020

    2020

  10. [18]

    Double q-learning

    Hasselt, H. Double q-learning. Advances in neural information processing systems, 23, 2010

    2010

  11. [19]

    Dropout q-functions for doubly efficient reinforcement learning

    Hiraoka, T., Imagawa, T., Hashimoto, T., Onishi, T., and Tsuruoka, Y. Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint arXiv:2110.02034, 2021

    2021 arXiv

  12. [20]

    S., Chang, R., Rybkin, O., and Jayaraman, D

    Hu, E. S., Chang, R., Rybkin, O., and Jayaraman, D. Planning goals for exploration. arXiv preprint arXiv:2303.13002, 2023

    2023 arXiv

  13. [21]

    Enhanced experience replay generation for efficient reinforcement learning

    Huang, V., Ley, T., Vlachou-Konchylaki, M., and Hu, W. Enhanced experience replay generation for efficient reinforcement learning. arXiv preprint arXiv:1705.08245, 2017

    2017 arXiv

  14. [22]

    Hunter, J. D. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9 0 (03): 0 90--95, 2007

    2007

  15. [23]

    An investigation of generative replay in deep reinforcement learning

    Imre, B. An investigation of generative replay in deep reinforcement learning. B.S. thesis, University of Twente, 2021

    2021

  16. [24]

    W., Parbhoo, S., and Ghassemi, M

    Killian, T. W., Parbhoo, S., and Ghassemi, M. Risk sensitive dead-end identification in safety-critical offline reinforcement learning. arXiv preprint arXiv:2301.05664, 2023

    2023 arXiv

  17. [25]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    2017 arXiv

  18. [26]

    CURL : Contrastive unsupervised representations for reinforcement learning

    Laskin, M., Srinivas, A., and Abbeel, P. CURL : Contrastive unsupervised representations for reinforcement learning. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research...

    2020

  19. [27]

    Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation

    Li, B., Tang, H., Zheng, Y., Hao, J., Li, P., Wang, Z., Meng, Z., and Wang, L. Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation. arXiv preprint arXiv:2109.05490, 2021

    2021 arXiv

  20. [29]

    P., Hunt, J

    Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  21. [30]

    Unlock the intermittent control ability of model free reinforcement learning

    Liu, J., Jianye, H., Hao, X., Ma, Y., ZHENG, Y., Hu, Y., and Lv, T. Unlock the intermittent control ability of model free reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

  22. [31]

    Liu, J., Ceron, J. S. O., Courville, A., and Pan, L. Neuroplastic expansion in deep reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=20qZK2T7fa

    2025

  23. [32]

    W., and Parker-Holder, J

    Lu, C., Ball, P., Teh, Y. W., and Parker-Holder, J. Synthetic experience replay. Advances in Neural Information Processing Systems, 36, 2024

    2024

  24. [33]

    Revisiting plasticity in visual reinforcement learning: Data, modules and training stages

    Ma, G., Li, L., Zhang, S., Liu, Z., Wang, Z., Chen, Y., Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. arXiv preprint arXiv:2310.07418, 2023

    2023 arXiv

  25. [34]

    Online reinforcement learning with uncertain episode lengths

    Mandal, D., Radanovic, G., Gan, J., Singla, A., and Majumdar, R. Online reinforcement learning with uncertain episode lengths. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 9064--9071, 2023

    2023

  26. [35]

    Tactical optimism and pessimism for deep reinforcement learning

    Moskovitz, T., Parker-Holder, J., Pacchiano, A., Arbel, M., and Jordan, M. Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 12849--12863, 2021

    2021

  27. [36]

    Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control

    Nauman, M., Ostaszewski, M., Jankowski, K., Mi o \'s , P., and Cygan, M. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint arXiv:2405.16158, 2024

    2024 arXiv

  28. [37]

    The primacy bias in deep reinforcement learning

    Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp.\ 16828--16847. PMLR, 2022

    2022

  29. [38]

    Obando-Ceron, J., Courville, A., and Castro, P. S. In value-based deep reinforcement learning, a pruned network is a good network. Architecture, 4: 0 4--5, 2024 a

    2024

  30. [39]

    Obando-Ceron, J., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J., Dziugaite, K., Precup, D., and Castro, P. S. Mixtures of experts unlock parameter scaling for deep rl. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 38520--38540, 2024 b

    2024

  31. [40]

    Oliphant, T. E. Python for scientific computing. Computing in Science & Engineering, 9 0 (3): 0 10--20, 2007. doi:10.1109/MCSE.2007.58

    2007 doi

  32. [41]

    Reinforcement learning with dynamic boltzmann softmax updates

    Pan, L., Cai, Q., Meng, Q., Chen, W., Huang, L., and Liu, T.-Y. Reinforcement learning with dynamic boltzmann softmax updates. arXiv preprint arXiv:1903.05926, 2019

    1903 arXiv

  33. [42]

    Softmax deep double deterministic policy gradients

    Pan, L., Cai, Q., and Huang, L. Softmax deep double deterministic policy gradients. Advances in neural information processing systems, 33: 0 11767--11777, 2020

    2020

  34. [43]

    Regularized softmax deep multi-agent q-learning

    Pan, L., Rashid, T., Peng, B., Huang, L., and Whiteson, S. Regularized softmax deep multi-agent q-learning. Advances in Neural Information Processing Systems, 34: 0 1365--1377, 2021

    2021

  35. [44]

    Time limits in reinforcement learning

    Pardo, F., Tavakoli, A., Levdik, V., and Kormushev, P. Time limits in reinforcement learning. In International Conference on Machine Learning, pp.\ 4045--4054. PMLR, 2018

    2018

  36. [45]

    A., and Darrell, T

    Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp.\ 2778--2787. PMLR, 2017

    2017

  37. [46]

    M., and Restelli, M

    Poiani, R., Metelli, A. M., and Restelli, M. Truncating trajectories in monte carlo reinforcement learning. In International Conference on Machine Learning, pp.\ 27994--28042. PMLR, 2023

    2023

  38. [47]

    M., and Restelli, M

    Poiani, R., Nobili, N., Metelli, A. M., and Restelli, M. Truncating trajectories in monte carlo policy evaluation: An adaptive approach. Advances in Neural Information Processing Systems, 36, 2024

    2024

  39. [48]

    Optimistic exploration even with a pessimistic initialisation

    Rashid, T., Peng, B., Boehmer, W., and Whiteson, S. Optimistic exploration even with a pessimistic initialisation. arXiv preprint arXiv:2002.12174, 2020

    2002 arXiv

  40. [49]

    Prioritized experience replay

    Schaul, T. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015

    2015 arXiv

  41. [50]

    Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp.\ 30365--30380. PMLR, 2023

    2023

  42. [51]

    S., and Evci, U

    Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pp.\ 32145--32168. PMLR, 2023

    2023

  43. [52]

    Sokar, G., Obando-Ceron, J., Courville, A., Larochelle, H., and Castro, P. S. Don't flatten, tokenize! unlocking the key to softmoe's efficacy in deep rl, 2025. URL https://arxiv.org/abs/2410.01930

    2025 arXiv

  44. [53]

    Revisiting the softmax bellman operator: New benefits and new perspective

    Song, Z., Parr, R., and Carin, L. Revisiting the softmax bellman operator: New benefits and new perspective. In International conference on machine learning, pp.\ 5916--5925. PMLR, 2019

    2019

  45. [54]

    Prioritizing samples in reinforcement learning with reducible loss

    Sujit, S., Nath, S., Braga, P., and Ebrahimi Kahou, S. Prioritizing samples in reinforcement learning with reducible loss. Advances in Neural Information Processing Systems, 36: 0 23237--23258, 2023

    2023

  46. [55]

    Safe exploration by solving early terminated mdp

    Sun, H., Xu, Z., Fang, M., Peng, Z., Guo, J., Dai, B., and Zhou, B. Safe exploration by solving early terminated mdp. arXiv preprint arXiv:2107.04200, 2021

    2021 arXiv

  47. [56]

    sunk costs

    Sweis, B. M., Abram, S. V., Schmidt, B. J., Seeland, K. D., MacDonald, A. W., Thomas, M. J., and Redish, A. D. Sensitivity to “sunk costs” in mice, rats, and humans. Science, 361 0 (6398): 0 178--181, 2018. doi:10.1126/science.aar8644. URL https://www.science.org/doi/abs/10.11...

    2018 doi

  48. [57]

    Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  49. [58]

    M., Ramirez-Pedraza, A., Chavez-Urbiola, E

    Terven, J., Cordova-Esparza, D. M., Ramirez-Pedraza, A., Chavez-Urbiola, E. A., and Romero-Gonzalez, J. A. Loss functions and metrics in deep learning. arXiv preprint arXiv:2307.02694, 2023

    2023 arXiv

  50. [59]

    H., Meyers, E

    Turpin, M. H., Meyers, E. A., Fugelsang, J. A., Friedman, O., and Bia ek, M. Sunk cost bias and withdrawal aversion. The American Journal of Bioethics, 19 0 (3): 0 57--59, 2019

    2019

  51. [60]

    Deep reinforcement learning with double q-learning

    Van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016

    2016

  52. [61]

    and Drake Jr, F

    Van Rossum, G. and Drake Jr, F. L. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995

    1995

  53. [62]

    S., and Krishnamurthy, A

    Wang, Y., Wang, R., Du, S. S., and Krishnamurthy, A. Optimism in reinforcement learning with generalized linear function approximation. arXiv preprint arXiv:1912.04136, 2019

    1912 arXiv

  54. [63]

    Drm: Mastering visual reinforcement learning through dormant ratio minimization

    Xu, G., Zheng, R., Liang, Y., Wang, X., Yuan, Z., Ji, T., Luo, Y., Liu, X., Yuan, J., Hua, P., Li, S., Ze, Y., Daum'e, H., Huang, F., and Xu, H. Drm: Mastering visual reinforcement learning through dormant ratio minimization. ArXiv, abs/2310.19668, 2023. URL https://api.semant...

    arXiv

  55. [64]

    Mastering visual continuous control: Improved data-augmented reinforcement learning

    Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021

    2021 arXiv

  56. [65]

    Sample efficient deep reinforcement learning via local planning

    Yin, D., Thiagarajan, S., Lazic, N., Rajaraman, N., Hao, B., and Szepesvari, C. Sample efficient deep reinforcement learning via local planning. arXiv preprint arXiv:2301.12579, 2023

    2023 arXiv

  57. [66]

    Scaling robot learning with semantically imagined experience

    Yu, T., Xiao, T., Stone, A., Tompson, J., Brohan, A., Wang, S., Singh, J., Tan, C., Peralta, J., Ichter, B., et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302.11550, 2023

    2023 arXiv

  58. [67]

    Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning

    Zheng, R., Wang, X., Sun, Y., Ma, S., Zhao, J., Xu, H., Daum \'e III, H., and Huang, F. Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  59. [68]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.