Pith. sign in

REVIEW 2 major objections 6 minor 4 cited by

Behaviour Suite for Reinforcement Learning

T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces bsuite, an open-source collection of small, targeted experiments that isolate core reinforcement learning capabilities such as memory and exploration, with automated analysis that makes agent evaluation reproducible.

desk verdict A genuinely useful, well-engineered benchmark paper; the memory-length scoring has a known confound the authors flag themselves, and it does not sink the contribution. read the letter →

arxiv 1908.03568 v3 pith:R24MKQRM submitted 2019-08-09 cs.LG cs.AIstat.ML

Ian Osband , Yotam Doron , Matteo Hessel , John Aslanides , Eren Sezener , Andre Saraiva , Katrina McKinney , Tor Lattimore
show 6 more authors
Csaba Szepesvari Satinder Singh Benjamin Van Roy Richard Sutton David Silver Hado Van Hasselt
This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords reinforcementlearningbenchmarkingexplorationmemorycreditassignmentreproducibilitydiagnosticexperimentsscalableevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning research needs a way to tell which of an agent's underlying capabilities—memory, exploration, credit assignment, robustness to noise and scale—is the bottleneck, and current large-scale game benchmarks mix all of these together. This paper argues that progress is better served by a suite of small, targeted 'unit test' experiments, each designed to isolate one capability and swept over problem size so that scaling behaviour becomes visible. To make that practical, it introduces bsuite, an open-source library that instantiates the experiments, fixes the interaction protocol for each, and automatically produces summary scores, plots, and a conference-ready report from any agent's logs. The paper demonstrates on two flagship experiments, memory length and deep sea, that the suite separates agents that have a capability from those that do not, and that scaling curves expose algorithmic limits that a single final-score comparison would hide.

What carries the argument

The load-bearing machinery is the notion of a 'bsuite experiment', which pins down three components: an environment family parameterized to vary difficulty smoothly (such as a T-maze of length $N=1,\dots,100$ exponentially spaced or a deep-sea grid of size $N=10,\dots,50$), a fixed interaction protocol (for example 10,000 episodes with automatic logging), and a fixed analysis that maps the log to a summary score in $[0,1]$. The summary score is normalized so that a random agent receives 0 and an optimal policy 1, with task-specific milestones such as 'average regret below 75% of random' for memory length; the same protocol also produces scaling curves showing how performance changes with the difficulty parameter. This three-part structure is what lets bsuite claim to isolate individual capabilities: because the environment, interaction, and analysis are fixed in advance, differences in score can be attributed to agent behaviour rather than to evaluation choices, and the parameter sweep makes scaling properties visible.

What would settle it

If a feedforward agent that cannot store information across timesteps scores above chance on the longest memory-length task, then the suite's claim to isolate memory is false.

Watch

Extended reading notes

Core claim

The central claim is that the most useful RL benchmarks need not be large or close to real-world problems; a small collection of carefully staged 'unit tests' can reveal which of an agent's core capabilities—memory, exploration, credit assignment, robustness to noise and scale, and basic learning—are present and how they scale. Each bsuite experiment specifies an environment family, a fixed number of episodes of interaction, and a fixed analysis that converts logged behaviour into a summary score in [0,1]. The paper shows on two flagship examples that this design distinguishes capable from incapable agents: on the memory-length T-maze only the recurrent actor-critic scores well, while on the deep-sea grid only Bootstrapped DQN, an algorithm built for deep exploration, scales to large problem sizes. Alongside the experiments, the open-source bsuite library automates running, logging, plotting, and report generation, so that the same evaluation can be applied to any agent with minimal integration effort.

Load-bearing premise

The central claim rests on the assumption that scoring well on a small, stripped-down task such as the T-maze or deep-sea grid genuinely reflects the targeted capability (memory or deep exploration) rather than an artifact of optimization details, reward scale, or other confounds.

Editorial extensions

If this is right

  • A researcher developing a new exploration algorithm can run bsuite in under an hour and see from the deep-sea sweep whether the alleged improvement is real or whether the bottleneck lies elsewhere.
  • The automated analysis produces a one-page LaTeX report that can be appended to a conference submission, making capability evaluation a standard part of RL papers.
  • Scale sweeps convert qualitative claims into empirical scaling curves, as when the recurrent actor-critic's memory cut-off matches its backprop-through-time length of 30 and Bootstrapped DQN's exploration improves gracefully as the grid grows to size 50.
  • Because the library accepts agents written for several common codebases, a bsuite result becomes a common currency for comparing agents across different implementations.
  • A standing committee will review community-submitted experiments through pull requests, so the suite can grow beyond its initial 2019 release.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same five design criteria (targeted, simple, challenging, scalable, fast) could be used to curate unit tests for capabilities the 2019 release omits, such as hierarchical credit assignment or multi-agent coordination.
  • A direct way to test the suite's isolation promise would be to measure, across many agents, the correlation between scores on different capabilities; near-zero correlations would support the claim that each task measures something distinct.
  • The hand-chosen score thresholds (e.g., regret below 75% of random for memory length) could be replaced with thresholds derived from seed statistics, which would make it possible to report uncertainty alongside each summary score.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces bsuite, an open-source suite of diagnostic reinforcement-learning experiments with automated evaluation and analysis. It states design criteria for the experiments, describes two flagship experiments in detail (memory length and deep sea), and reports baseline results for DQN, A2C, and Bootstrapped DQN. The central claim is that bsuite provides clear, informative, and scalable problems that isolate key RL capabilities and enable reproducible comparison of agents, in the spirit of an 'MNIST for RL'.

Significance. If the stated design goals are met, bsuite would be a valuable community resource: it offers targeted experiments with smooth scaling sweeps, a standardized interaction protocol, and low-cost evaluation, complementing large-scale benchmarks such as Atari. The open-source implementation, reference algorithms, and automated LaTeX report generation are concrete reproducibility contributions, and the two worked examples demonstrate the intended workflow. The baseline results are also informative, for example showing that Bootstrapped DQN, but not DQN or A2C, scales gracefully on deep sea. The main caveat is that the flagship memory-length experiment does not fully isolate memory as claimed.

major comments (2)
  1. [Section 2.1 (memory length) and Sections 1.4/2 (design claims)] The experiment varies N, which changes not only the delay between context and reward but also the credit-assignment/discounting horizon and, for recurrent agents, the BPTT truncation horizon. The paper's own Figure 2b exposes the confound: A2C's sharp cutoff at N≈30 coincides with the BPTT length of 30, so a perfect-memory agent trained with BPTT-30 would fail the same way for N>30. This contradicts the claim that bsuite experiments are 'targeted unit tests' that 'remove all confounds' from the capability of interest. The score and the label 'memory length' therefore overstate the isolation of memory. I recommend either renaming/reframing the experiment as a joint test of memory and optimization horizon, adding a fixed-horizon control that varies only the memory delay, or both. The suite remains useful as a comparative diagnostic, but this load-bearing design claim needs revision.
  2. [Section 2.2 vs Appendix A.4.1] The summary score for deep sea is defined inconsistently between the two locations. Section 2.2 states that the score is the percentage of runs for which average regret drops below 0.9 faster than the 2^N episodes expected by dithering, whereas Appendix A.4.1 defines it as the percentage of runs with average regret below 90% of the random policy's regret. These are different criteria (one is time-relative, the other is a fixed threshold). Since the summary scores are the paper's primary quantitative output, the definition should be stated once, precisely, and used consistently everywhere.
minor comments (6)
  1. [Section 2.2] The expressions '2 − N' and '2 N episodes' are missing superscripts; they should read 2^{-N} and 2^N.
  2. [Section 2.1 / Appendix A.6.1] The memory-length protocol is described with 10k episodes in Section 2.1 but with 1k episodes in Appendix A.6.1; these should be reconciled.
  3. [Section 4] There is a typo in the sentence 'Since the environments handle the logging themselves, your don't need any additional logging'; it should be 'you don't need'.
  4. [Figures 2b and 4b] The captions should state what quantity is plotted on each axis and what the dashed reference line corresponds to beyond 'for reference'; currently the reader must infer this from the text.
  5. [References] The reference list contains duplicate entries for the Nature DQN paper as Mnih et al. 2015a and 2015b; one entry should be removed and citations reconciled.
  6. [Sections 2.1 and 2.2] The threshold values used in the summary scores (75% of random regret for memory length, 90% for deep sea) appear arbitrary; a sentence justifying their choice or showing sensitivity would improve interpretability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: bsuite is a measurement benchmark, not a derivation, and the paper's self-citations are not load-bearing.

full rationale

The paper makes no derived prediction that could reduce to its own inputs. It introduces a set of diagnostic environments, interaction protocols, and analyses (Section 2), and agent scores are measurements obtained by running open-source implementations, not quantities derived from the task definitions. The phrase 'By construction, an agent that performs well on this task has mastered some use of memory over multiple timesteps' (Section 2.1) is a construct-validity claim about the experiment, not a circular derivation; the paper explicitly explains the A2C cutoff via the agent's backprop-through-time length of 30, which is an empirical observation about the agent rather than an input to the task. Self-citations to Bootstrapped DQN (Osband et al., 2016; 2018) and deep exploration (Osband et al., 2017) are used as baseline agent definitions and background, not as load-bearing justification for the suite's central claims, and the exploration requirement is independently grounded in external references such as Kearns and Singh (2002) and Jaksch et al. (2010). The paper even flags its own limitation, the lack of targeted HRL experiments, in a footnote in Section 3, which further confirms that it is not asserting a closed derivation. Since no step in the paper reduces a claimed result to its own definition, a fitted parameter renamed as a prediction, or a self-citation chain, the appropriate finding is no circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The bsuite paper introduces no new physical or mathematical entities. Its free parameters are design choices for thresholds, sizes, and episode counts, which are hand-picked but transparent. The key axioms are domain assumptions about what constitutes core RL capabilities and how to measure them. The suite itself is a software artifact, not an invented entity in the scientific sense.

free parameters (3)
  • memory_length score threshold = 75% of random policy regret
    The summary score for the memory length experiment is defined as the percentage of runs where average regret is less than 75% of that of a uniformly random policy. This threshold is chosen by the authors to define success, and different thresholds could change the score.
  • deep_sea score criterion = average regret below 0.9 faster than 2^N episodes
    The deep sea experiment's summary score is the percentage of runs where average regret drops below 0.9 faster than the 2^N episodes expected by random dithering. This is an arbitrary but reasonable criterion.
  • experiment sizes and episode counts = N=1..100 for memory, N=10..50 for deep sea, 10k episodes for most experiments
    The specific ranges of problem sizes and the number of interaction episodes are design choices that balance compute time and signal. They are not fitted to any external data but affect how well the experiments distinguish agents.
assumptions (3)
  • domain assumption An efficient RL agent must address generalization, exploration, and long-term consequences simultaneously (Section 1).
    This framing underlies the choice of experiments in bsuite. It is a standard view in RL, but it is assumed rather than proven that these three challenges are the core ones.
  • domain assumption A good bsuite experiment must be targeted, simple, challenging, scalable, and fast (Section 2).
    These five qualities are the criteria for including experiments, chosen by the authors. They are not derived from first principles, but they are stated explicitly and are reasonable for a benchmark suite.
  • domain assumption Performance on small diagnostic tasks provides insight into an agent's performance on larger real-world problems (Sections 1 and 3).
    The entire value proposition of bsuite rests on this assumption, which is stated as a hope rather than demonstrated. The paper does not provide evidence that bsuite scores correlate with performance on Atari or control tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Behaviour Suite for Reinforcement Learning." pith.science (2026). https://pith.science/paper/R24MKQRM

@misc{pith2026190803568,
  author       = {Pith},
  title        = {Pith review of: Behaviour Suite for Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R24MKQRM}},
  note         = {Machine review of arXiv:1908.03568}
}
read the original abstract

This paper introduces the Behaviour Suite for Reinforcement Learning, or bsuite for short. bsuite is a collection of carefully-designed experiments that investigate core capabilities of reinforcement learning (RL) agents with two objectives. First, to collect clear, informative and scalable problems that capture key issues in the design of general and efficient learning algorithms. Second, to study agent behaviour through their performance on these shared benchmarks. To complement this effort, we open source github.com/deepmind/bsuite, which automates evaluation and analysis of any agent on bsuite. This library facilitates reproducible and accessible research on the core issues in RL, and ultimately the design of superior learning algorithms. Our code is Python, and easy to use within existing projects. We include examples with OpenAI Baselines, Dopamine as well as new reference implementations. Going forward, we hope to incorporate more excellent experiments from the research community, and commit to a periodic review of bsuite from a committee of prominent researchers.

Figures

Figures reproduced from arXiv: 1908.03568 by the authors.

Figure 1
Figure 1. Illustration of the ‘memory length’ environment [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Selected output from bsuite evaluation on ‘memory length’. consequences of its actions towards cumulative rewards, an agent seeking to ‘explore’ must consider how its actions can position it to learn more effectively in future timesteps. The literature on efficient exploration broadly states that only agents that perform deep explo￾ration can expect polynomial sample complexity in learning (Kearns & Singh, 2002). Th… view at source ↗
Figure 3
Figure 3. Deep-sea exploration: a simple example where deep exploration is critical. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Selected output from bsuite evaluation on ‘deep sea’. 3 How to use bsuite This section describes some of the ways you can use bsuite in your research and develop￾ment of RL algorithms. Our aim is to present a high-level description of some research and engineering use …
Figure 5
Figure 5. Figure 5: We aggregate experiment performance with a snapshot of 7 core capabilities. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: A snapshot of agent behaviour [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 8
Figure 8. Figure 8: A snapshot of agent behaviour [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 10
Figure 10. Figure 10: A snapshot of agent behaviour [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. T-GRAB: A Synthetic Diagnostic Benchmark for Learning on Temporal Graphs

    cs.LG 2025-07 conditional novelty 7.0 of 10

    T-GRAB, a set of three synthetic temporal-graph tasks, shows that no current TGNN reliably does counting, delayed cause-effect, or long-range spatio-temporal reasoning.

  2. Octax: Accelerated CHIP-8 Arcade Environments for Reinforcement Learning in JAX

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Octax is a JAX-based CHIP-8 emulator that runs thousands of parallel arcade environments on GPUs (350k steps/s) and supports LLM-generated games for RL training.

  3. On the Effect of Regularization in Policy Mirror Descent

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A large empirical sweep shows that in Policy Mirror Descent, MDP and Drift regularizers are partly substitutable, yet their precise combination determines temperature robustness.

  4. OpenSpiel: A Framework for Reinforcement Learning in Games

    cs.LG 2019-08 accept novelty 6.0 of 10

    OpenSpiel provides a unified, open-source API for coding, running, and evaluating many game types and algorithms for reinforcement learning and game theory in one framework.

Reference graph

Works this paper leans on

18 extracted references · 6 canonical work pages · cited by 4 Pith papers

  1. [5]

    Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G

    URL http://arxiv.org/abs/ 1606.01540. Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning

    arXiv

  2. [10]

    The gains are most extreme in the exploration tasks, where ensemble sizes less than 10 are not able to solve large ‘deep sea’ tasks, but larger ensembles solve them reliably

    These results are not predicted by the theoretical scaling of proven bounds (Lu & Van Roy, 2017), but are consistent with previous empirical findings (Osband et al., 2017; Russo et al., 2017). The gains are most extreme in the exploration tasks, where ensemble sizes less than 10 are not able to solve large ‘deep sea’ tasks, but larger ensembles solve them ...

    work page 2017

  3. [11]

    Revisiting the Arcade Learning Environment: Evaluation protocols and open problems for general agents

    Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the Arcade Learning Environment: Evaluation protocols and open problems for general agents. arXiv preprint arXiv:1709.06009 ,

    arXiv

  4. [13]

    Deep Exploration via Randomized Value Functions

    Ian Osband, Daniel Russo, Zheng Wen, and Benjamin Van Roy. Deep exploration via randomized value functions. arXiv preprint arXiv:1703.07608 ,

    work page Pith review arXiv

  5. [17]

    doi: 10.1126/science.aar6404

    ISSN 0036-8075. doi: 10.1126/science.aar6404. URL https://science.sciencemag.org/content/362/6419/1140. Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences , 74(8):1309–1331,

    doi:10.1126/science.aar6404

  6. [20]

    12 Published as a conference paper at ICLR 2020 A Experiment summary This appendix outlines the experiments that make up the bsuite 2019 release

    URL http://arxiv.org/abs/1907.02057. 12 Published as a conference paper at ICLR 2020 A Experiment summary This appendix outlines the experiments that make up the bsuite 2019 release. In the interests of brevity, we provide only an outline of each experiment here. Full documentation for the environments, interaction and analysis are kept with code at githu...

    arXiv 1907

  7. [1952]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings,

    2015

  8. [1958]

    A tutorial on Thompson sampling

    Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, and Ian Osband. A tutorial on Thompson sampling. arXiv preprint arXiv:1707.02038 ,

    arXiv

Show all 18 references
  1. [1961]

    MIPLIB 2017,

    miplib2017. MIPLIB 2017,

    2017

  2. [1983]

    Deepmind lab.arXiv preprint arXiv:1612.03801,

    Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich K¨ uttler, Andrew Lefrancq, Simon Green, V´ ıctor Vald´ es, Amir Sadik, et al. Deepmind lab.arXiv preprint arXiv:1612.03801,

    arXiv

  3. [2007]

    doi: 10.1109/ MCSE.2007.53

    ISSN 1521-9615. doi: 10.1109/ MCSE.2007.53. URL https://ipython.org. Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386,

    2007

  4. [2009]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 ,

    arXiv

  5. [2013]

    Large-scale machine learning with stochastic gradient descent

    L´ eon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT’2010, pp. 177–186. Springer,

    2010

  6. [2015]

    10 Published as a conference paper at ICLR 2020 Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton

    URL http://arxiv.org/abs/1412.6980. 10 Published as a conference paper at ICLR 2020 Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems 25, pp. 1097–1105,

    2020 arXiv

  7. [2016]

    Reconciling modern machine learning and the bias-variance trade-off

    9 Published as a conference paper at ICLR 2020 Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning and the bias-variance trade-off. arXiv preprint arXiv:1812.11118 ,

    2020 arXiv

  8. [2017]

    org/abs/1709.06560

    URL http://arxiv. org/abs/1709.06560. Alexey Grigorevich Ivakhnenko. The group method of data of handling; a rival of the method of stochastic approximation. Soviet Automatic Control, 13:43–55,

    arXiv

  9. [2018]

    org/abs/1812.06110

    URL http://arxiv. org/abs/1812.06110. Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273–297,

    arXiv

  10. [2019]

    11 Published as a conference paper at ICLR 2020 Fernando P´ erez and Brian E. Granger. IPython: a system for interactive scientific computing. Computing in Science and Engineering , 9(3):21–29, May

    2020

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.