REVIEW 4 major objections 5 minor 11 references
Adaptive Action Duration with Contextual Bandits for Deep Reinforcement Learning in Dynamic Environments
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper proposes adding a contextual bandit to DQN to learn how many frames each action should be repeated, and reports that this adaptive duration selection beats static and discrete repetition baselines on three Atari games.
desk verdict The bandit-based duration selection is a plausible incremental idea, but the empirical section as printed is not credible: Table 2's percentages sum to over 100% for three games, which is impossible for a distribution. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a contextual bandit module attached to DQN that outputs a probability distribution over durations $d \in \{1,\dots,d_{\max}\}$ conditioned on the state, with $d_{\max}=20$. It is trained by policy gradient on the reward $r_b = Q(s_{t+d_t}, a'; \theta) - Q(s_t, a_t; \theta)$, the change in the DQN's predicted Q-value after executing action $a_t$ for $d_t$ frames. This Q-difference signal is what lets the bandit learn which durations are useful in different states without external labels, and it is the mechanism that carries the argument from architecture to reported performance.
What would settle it
One concrete test is to train the same architecture but replace the bandit reward with the actual return difference $r_t + \gamma \max_a Q(s_{t+d_t}, a) - Q(s_t, a_t)$; if the score gains vanish or reverse, the Q-difference signal is the source of the improvement. A second, simpler falsifier is to run the full method but sample durations uniformly at random from the same learned marginal distribution; if performance does not drop, then the state-dependent duration choices are not doing the work.
Extended reading notes
Core claim
The central claim is that a contextual bandit can learn action durations and that this adaptive repetition improves over fixed and discrete repetition baselines. The paper demonstrates this in Atari 2600 games, where Bandit-DQN achieves higher average episode scores than DQN with ARR=4, DQN with ARR=20, and DFDQN in Seaquest, Space Invaders, and Enduro. The paper interprets the learned duration distributions as evidence of game-specific temporal strategies, such as short durations in Space Invaders for quick reflexes and long durations in Enduro for sustained driving, explaining the performance gains.
Load-bearing premise
The bandit is trained on the change in the DQN's own Q-values, and this signal is assumed to be a reliable indicator of how good a chosen duration is, but during early training these Q-values are inaccurate and the bandit may chase estimation errors instead of true returns.
Editorial extensions
If this is right
- If the reported gains hold, adaptive duration selection is a lightweight extension to DQN that improves scores while potentially reducing computation by repeating actions that do not need fine-grained control.
- The learned duration distributions imply that agents can acquire game-specific temporal policies, such as short bursts for shooting and long stretches for driving, without explicit supervision.
- The approach extends Dynamic Action Repetition by replacing a discrete set of repetition rates with a range of durations, which may transfer to tasks where optimal durations vary continuously.
- The method is directly compatible with other DQN variants and could be ported to policy-based algorithms, as the paper suggests for A3C.
Reading between the lines
- The Q-difference reward is a self-referential signal: early in training, when Q-values are inaccurate, the bandit may reinforce estimates that are wrong in the same way, so the reported improvements could partly stem from exploiting Q-value noise rather than true temporal benefits.
- The Q*Bert result is an outlier: the score jumps from 1150 (DFDQN) to 3500, far larger than gains in other games and not explained by the modest duration distribution shift, suggesting a possible evaluation variance or a non-temporal mechanism.
- A testable extension is to replace the bandit reward with the actual return difference $r_t + \gamma \max_a Q(s_{t+d}, a) - Q(s_t, a_t)$; if the method's advantage disappears, the Q-value-change signal is the critical component, not the adaptivity itself.
- The discrete duration set (1 to 20) is still a coarse approximation of continuous time; extending the bandit to output a continuous duration or to modulate the frame count mid-action could be a natural next step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Bandit-DQN, a DQN extension that uses a contextual bandit to select an action duration d in {1,...,d_max} for every selected action, with the bandit trained by policy gradient on a reward equal to the change in the DQN's own Q-values. The method is evaluated on five Atari 2600 games against static action repetition rates and a dynamic frameskip baseline (DFDQN), and the authors claim significant performance improvements. The central contribution is empirical: an adaptive duration selection mechanism that is claimed to outperform static and dynamic baselines on Seaquest, Space Invaders, Enduro, Alien, and Q*Bert.
Significance. If the empirical claims were properly supported, adaptive action-duration selection would be a useful direction for improving sample efficiency and temporal control in deep RL, and the contextual-bandit formulation is a natural extension of dynamic action repetition. However, the paper provides no theoretical analysis, no code, no error bars or multiple seeds, and the reported duration statistics in Table 2 are internally inconsistent. The main evidence for the headline claim is therefore not credible as presented. The paper does identify a relevant problem and proposes a simple, potentially useful architecture; those strengths are acknowledged, but they do not compensate for the current lack of reliable experimental support.
major comments (4)
- [Section 4.2, Table 2] The three duration categories Short (1-5), Medium (6-7), and Long (8-11) partition the integer durations 1-11, so the percentages for each game must sum to 100% within rounding error. However, Seaquest sums to 54+30+26=110%, Enduro to 34+25+45=104%, and Q*Bert to 49+30+31=110%. This is mathematically impossible for a distribution over disjoint categories and indicates that the reported statistics were not actually computed from the agent's behavior. The table also omits the range 12-20 entirely, even though d_max=20, with no explanation. Because this analysis table is used to explain the performance gains, its unreliability directly undermines the empirical claims in Table 1.
- [Section 4.1, Table 1] The Q*Bert row reports Bandit-DQN at 3500 versus DFDQN at 1150, a 204% improvement that is wildly larger than the improvements in any other game. No mechanism for this outlier is given, no error bars or standard deviations are reported, and the number is consistent with a typo. Since Table 1 is the only evidence for the central claim that Bandit-DQN outperforms all baselines, the authors must provide raw per-seed results, standard deviations, and statistical tests across at least five independent runs before the claim can be evaluated.
- [Section 3.1, Eq. (1)] The bandit reward rb = Q(s_{t+dt}, a'; theta) - Q(s_t, a_t; theta) is a difference of Q-value estimates from the same network that is being trained. This objective is not the environment reward, is not discounted, and has no theoretical guarantee of correlating with true return. During early training Q-values are inaccurate, and because theta changes simultaneously, the policy-gradient update on the bandit is biased with respect to any fixed reward signal. The paper provides no ablation comparing this heuristic with alternatives (e.g., using the environment reward accumulated over the duration), so the reported improvements could result from exploiting Q-value estimation error rather than from better temporal control. This is a load-bearing assumption and needs either a proof of correctness or an ablation that separates the effect of the reward signal from the effect of adaptive durations.
- [Section 4, Experimental protocol] The text states that the authors 'report the best testing epoch score.' Choosing the maximum testing score across training epochs is a form of test-set selection that can substantially inflate reported performance and makes comparisons across methods unreliable, especially without multiple seeds. Combined with the absence of error bars, statistical tests, and a fixed evaluation protocol, the claim that 'Bandit-DQN outperforms all baselines' is not statistically substantiated. The authors should report the mean and standard deviation over seeds, ideally with a fixed policy snapshot or a final-policy evaluation.
minor comments (5)
- [Section 3.1] The notation 'd∈Z+' should be replaced with 'd ∈ {1,...,d_max}' to match the actual arm set, and the probability notation should be typeset as π_b(d_t | s_t; θ_b).
- [Algorithm 1, line 10] The transition stored in the replay memory includes d_t, but the DQN update described on line 11 does not use d_t in any way. Please clarify how the chosen duration affects the Q-learning loss, or remove d_t from the stored tuple if it is not used.
- [Abstract and Section 2] The abstract says the method offers 'finer-grained temporal control without predefined rates,' but the durations are still drawn from a discrete set {1,...,20}. Please rephrase to avoid implying a continuous range.
- [Section 2] The sentence 'Our work is the first to apply contextual bandits to action duration selection in DRL' is a strong novelty claim without a comprehensive literature survey. Please soften it or provide more evidence that no prior work exists.
- [References] Several references use 'and others' instead of listing authors, and reference [8] is described only by a title with no clear venue. Please complete the reference information; also, [5] is a news-recommendation paper and is not directly an RL method, so consider citing more relevant contextual-bandit RL literature.
Circularity Check
No significant circularity: final performance is evaluated on environment returns, and the bandit's Q-difference reward is an internal critic signal, not a self-definitional reduction.
full rationale
The paper's derivation chain is not circular in the analyzer's sense. The bandit is trained with the reward r_b = Q(s_{t+dt}, a';θ) − Q(s_t, a_t;θ), which is an internal surrogate derived from the DQN's own Q-values. However, the DQN is itself trained on environment transitions via the standard DQN loss, and the paper's central performance claims (Table 1) are measured on external episode scores, not on r_b. Thus the bandit does not predict a quantity that is defined in terms of its own output; it optimizes a critic-like signal that is only indirectly connected to the true objective. This is an actor-critic-style architecture rather than a self-definitional equivalence. There are no load-bearing self-citations: the cited DAR [4] and DQN [6] are external prior works, and no uniqueness theorem is imported from the authors. I separately note a data-integrity problem that is not circularity: Table 2's duration percentages sum to over 100% for Seaquest (110%), Enduro (104%), and Q*Bert (110%), making the reported duration distribution physically impossible; this undermines the empirical analysis but does not constitute a circular derivation. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- d_max (maximum action duration) =
20
assumptions (3)
- ad hoc to paper Q-value difference rb is a valid reward for duration selection
- standard math Standard MDP and policy gradient framework
- domain assumption Atari 2600 environment serves as a representative testbed
Cite this review
Pith. "Pith review of Adaptive Action Duration with Contextual Bandits for Deep Reinforcement Learning in Dynamic Environments." pith.science (2026). https://pith.science/paper/XONAHMGM
@misc{pith2026250700030,
author = {Pith},
title = {Pith review of: Adaptive Action Duration with Contextual Bandits for Deep Reinforcement Learning in Dynamic Environments},
year = {2026},
howpublished = {\url{https://pith.science/paper/XONAHMGM}},
note = {Machine review of arXiv:2507.00030}
}
read the original abstract
Deep Reinforcement Learning (DRL) has achieved remarkable success in complex sequential decision-making tasks, such as playing Atari 2600 games and mastering board games. A critical yet underexplored aspect of DRL is the temporal scale of action execution. We propose a novel paradigm that integrates contextual bandits with DRL to adaptively select action durations, enhancing policy flexibility and computational efficiency. Our approach augments a Deep Q-Network (DQN) with a contextual bandit module that learns to choose optimal action repetition rates based on state contexts. Experiments on Atari 2600 games demonstrate significant performance improvements over static duration baselines, highlighting the efficacy of adaptive temporal abstractions in DRL. This paradigm offers a scalable solution for real-time applications like gaming and robotics, where dynamic action durations are critical.
Reference graph
Works this paper leans on
-
[1]
Marc G. Bellemare and others. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253--279, 2013
work page 2013
-
[2]
Frame skip is a powerful parameter for learning to play Atari
Alexander Braylan and others. Frame skip is a powerful parameter for learning to play Atari. In AAAI Workshop on Learning for General Competency in Video Games, 2015
work page 2015
-
[3]
Daniel T. Gilbert and Timothy D. Wilson. Prospection: Experiencing the future. Science, 317(5843):1351--1354, 2007
work page 2007
-
[4]
Aravind S. Lakshminarayanan and others. Dynamic action repetition for deep reinforcement learning. In AAAI Conference on Artificial Intelligence, pages 2133--2139, 2017
work page 2017
-
[5]
A contextual-bandit approach to personalized news article recommendation
Lihong Li and others. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the World Wide Web Conference, pages 661--670, 2010
work page 2010
-
[6]
Human-level control through deep reinforcement learning
Volodymyr Mnih and others. Human-level control through deep reinforcement learning. Nature, 518(7540):529--533, 2015
work page 2015
-
[7]
Asynchronous methods for deep reinforcement learning
Volodymyr Mnih and others. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pages 1928--1937, 2016
work page 1928
-
[9]
Mastering the game of Go with deep neural networks and tree search
David Silver and others. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484--489, 2016
work page 2016
Show all 11 references
-
[10]
Sutton and others
Richard S. Sutton and others. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):181--211, 1999
1999
-
[11]
Temporal abstraction in reinforcement learning with the successor representation
Mohsen Vafadost and others. Temporal abstraction in reinforcement learning with the successor representation. arXiv preprint arXiv:1310.0713, 2013
2013 arXiv
-
[12]
Ԅ; ˏ 9"j8 6 tzyy l6@ kp' 7l6 d2X Q q Gr P m6o&EQo>qD0 dY ٗ_ ڵk (Y ` ^ݻw xSSS:j|Ǐ|YUnKh0 `` 歷2 ap8 4M˲o 9r xUUj H / Q eCP4 T < _ \ Ȋ p8l0 X t:L&F oQw 7 h a&6c(ʲl27 D
11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEco...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.