Pith. sign in

REVIEW 3 major objections 5 minor 69 references

Dynamic Reinforcement Learning for Actors

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A per-neuron sensitivity rule steers a chaotic actor loop, replacing exploration noise and backpropagation through time; on two dynamic tasks it matches BPTT-trained success and adapts faster after environmental changes.

desk verdict SRL is a real novelty and the ablations are decent, but the paper's central local-to-global control claim is under-supported; send it to review, not desk reject. read the letter →

arxiv 2502.10200 v1 pith:IZRA3KFQ submitted 2025-02-14 cs.LG cs.AIcs.NE

Katsunari Shibata This is my paper · ORCID
classification cs.LGcs.AIcs.NE MSC 68T0768T0537D45
keywords dynamicreinforcementlearningsensitivitychaoticdynamicsrecurrentneuralnetworkexplorationactor-criticbackpropagationthroughtimeexploration-exploitationbalance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a qualitative shift in reinforcement learning: instead of training the actor's pointwise outputs, learn to control the dynamics of the actor network coupled to its environment. The actor is initialized with large recurrent weights so that the agent-environment loop is chaotic, making exploration deterministic and part of motion generation rather than external noise. Learning then adjusts a per-neuron quantity, the sensitivity of output to input, by two local rules: Sensitivity Adjustment Learning (SAL) maintains enough chaos to keep learning active, while Sensitivity-controlled Reinforcement Learning (SRL) lowers sensitivity around transitions followed by positive TD error and raises it around negative TD error. On a memory-dependent navigation task and a dynamic pattern-generation task, the author reports that this actor alone matches the success rate of a conventional BPTT-trained actor-critic with sensitivity maintenance, learns with one-eighth to one-fifth of the actor's computation, and re-adapts faster after the environment changes. If correct, the result would mean that reinforcement can shape global system dynamics from purely local neuron updates, a step toward the author's hypothesis that exploration can grow into thinking.

What carries the argument

The load-bearing object is the sensitivity index of Eq. (4), $s(U;w)=f'(U)\lVert w\rVert$, which measures how much a small perturbation of the input neighborhood is expanded or contracted by the neuron's processing. Under the hyperbolic tangent nonlinearity the sensitivity updates reduce to the fully local rules $\Delta w_t = -\eta_{\mathrm{SRL}}(\Delta t/\tau)\hat{r}_t(1-o_t^2)(w/\lVert w\rVert - 2 o_t \lVert w\rVert x_t)$ and $\Delta \theta_t = 2\eta_{\mathrm{SRL}}(\Delta t/\tau)\hat{r}_t o_t (1-o_t^2)\lVert w\rVert$, with SAL using the same gradients in the opposite direction when the moving-average sensitivity falls below a threshold. These formulas are the mechanism: they convert a scalar TD error into a state- and direction-dependent adjustment of local contraction or expansion, and through the chain of neurons into control of the global Lyapunov behavior of the agent-environment loop.

What would settle it

Measure the actual expansion of a tiny input perturbation through a single dynamic neuron with $\tau=5$ when its input is its own delayed output, and compare the measured ratio $|\Delta o|/\lVert \Delta x\rVert$ with the formula $f'(U)\lVert w\rVert$ over many steps; if the ratio deviates systematically from the formula on recurrent inputs, the sign-controlled update would be adjusting the wrong quantity.

Watch

Extended reading notes

Core claim

The central claim is that an actor RNN can learn dynamic, memory-dependent behavior without stochastic action selection and without backpropagation through time. What is controlled is the local convergence or divergence of state-space flow, quantified in each neuron by the sensitivity $s(U;w)=f'(U)\lVert w\rVert$, the Euclidean norm of the output gradient with respect to the neuron's input vector. The update equations (16)-(17) make a positive TD error reduce the neuron's sensitivity, rendering a good state transition reproducible, and a negative TD error increase sensitivity, restoring exploratory irregularity. In two tasks, a sequential navigation task requiring the agent to remember whether it has passed a subgoal and a slider-crank control task requiring periodic force generation, the proposed actor-critic is reported to reach success rates comparable to the conventional BPTT actor-critic with SAL (36 successes in 40 runs in navigation; the highest success count in slider-crank), while spending roughly one-eighth and one-fifth of the actor-network learning computation, and to recover markedly faster after the environment changes. The author reads the resulting shift from irregular to rational state transitions as the core evidence for the hypothesis that exploration and thinking lie on a single chaotic-dynamics continuum.

Load-bearing premise

The paper's update rule treats the sensitivity of a dynamic recurrent neuron as the static gradient $f'(U)\lVert w\rVert$, which is justified only if the neuron's input varies slowly compared with its time constant $\tau$; recurrent inputs are previous outputs and can change quickly, so this slow-input assumption is the point where the local index could stop measuring true convergence or divergence.

Editorial extensions

If this is right

  • Actor learning cost per step drops to a fraction of the BPTT baseline because no temporal backpropagation and no random-number generator are required.
  • After an environmental change, negative TD errors automatically raise sensitivity, so exploratory behavior resumes quickly without any explicit reset or schedule.
  • SAL is load-bearing for both Dynamic RL and conventional BPTT: removing it lowers success rates, so maintaining per-neuron sensitivity is a general requirement for recurrent policy learning, not only for chaos-based learning.
  • Because exploration is embedded in the actor output and cannot be switched off, test-time performance remains slightly worse and more irregular than a noise-free conventional actor.
  • Applying SRL to hidden-to-hidden recurrent weights is what makes learning work; earlier chaos-based methods that left these weights unchanged did not improve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the slow-input assumption is the real constraint, the natural stress test is to compare the local sensitivity gradient with a numerically computed Jacobian on recurrent paths; this could determine how much the method depends on large time constants.
  • The same sensitivity argument could be ported to the critic network, which the paper leaves conventional; a dynamic critic that adjusts value-neuron sensitivity by TD error might stabilize value learning the way SRL stabilizes the actor.
  • The two tasks suggest a scaling prediction: deeper or sparser-reward environments should select for larger target sensitivity thresholds, since exploration needs stronger chaoticity early in learning.
  • The hypothesis that exploration grows into thinking predicts a measurable signature: as training progresses, trajectory divergence should remain positive but shrink, and after reward reversal it should spike; the exploration exponent computed in this paper is the natural observable for that prediction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes 'Dynamic Reinforcement Learning for Actors,' in which an actor RNN is made chaotic and learns by locally adjusting each neuron's sensitivity s=f'(U)||w|| rather than by backpropagating errors through time. A sensitivity adjustment learning (SAL) rule keeps sensitivity above a threshold to preserve chaotic exploration, while a sensitivity-controlled RL (SRL) rule increases or decreases sensitivity according to the sign of the TD error. The method is tested on a sequential navigation task and a slider-crank control task, with comparisons to BPTT-based actor-critic learning, ablations of SAL and SRL, and measurements of an 'exploration exponent.' The paper reports comparable or better success rates, faster adaptation after environmental changes, and lower computational cost, and it connects the results to speculative claims about exploration, thinking, and existential risk.

Significance. If the central mechanism were established, the paper would offer a genuinely interesting local-learning alternative to BPTT for recurrent actor networks, eliminating backward computation through time and external exploration noise while retaining comparable task performance. The update rules in Eqs. (9)-(10) and (16)-(17) are derived arithmetically in a transparent way, and the simulations use 40 runs with explicit success/failure criteria and targeted ablations that show both SAL and SRL contribute to performance. Credit is also due for the honest enumeration of remaining problems in Section 4.1. However, the load-bearing claim that locally updating per-neuron sensitivity reliably controls global convergence/divergence of the closed-loop agent-environment system is not supported by a derivation or by sign-conditioned empirical evidence. The dynamic-neuron sensitivity is introduced under an unverified slow-input assumption, and the reported exploration exponent is an aggregate over episodes rather than a measure of transitions grouped by TD-error sign. These gaps make the paper a promising proposal rather than a fully established result.

major comments (3)
  1. [§2, Eqs. (1), (4), (6)] The sensitivity defined in Eq. (4) is the norm of the gradient of o_t with respect to x_t for a static neuron. For the dynamic neuron in Eq. (1), the one-step Jacobian is (Delta_t/tau) f'(U_t) w, so the sensitivity is (Delta_t/tau) f'(U_t) ||w||, not f'(U_t) ||w|| as used in Eq. (4). The paper invokes a slow-input assumption, but recurrent feedback inputs are outputs at time t-1 and need not vary slowly relative to tau. Moreover, the moving average sbar in Eq. (6) and the SAL/SRL switching condition in Fig. 4 use the unscaled static value, which for the upper hidden layer with tau=5 overstates sensitivity by a factor of 5 relative to the actual one-step sensitivity. This scale mismatch can place neurons in the wrong learning regime, and the manuscript should either define a correct dynamic sensitivity or demonstrate empirically that the mismatch does not affect the claimed mechanism.
  2. [§1.4, §4.1, Fig. 13] No theoretical or controlled empirical bridge is provided from per-neuron sensitivity to the global contraction/expansion of the recurrent agent-environment system. In a nonlinear recurrent system, local convergence or divergence is governed by products and alignments of local Jacobians, so a scalar per-neuron index does not by itself determine the spectral norm of the global state-transition Jacobian. The exploration exponent defined in Eq. (24) and plotted in Fig. 13 is an aggregate average over episodes, not conditioned on the sign of the TD error. Consequently, the data do not demonstrate that positive-TD updates specifically increase reproducibility or that negative-TD updates specifically increase divergence; the observed decrease of the exponent during learning could also arise from SAL, output-layer regularization, or other side effects of weight changes. The authors should add sign-conditioned measurements of sensitivity or local divergence, or an analytical statement relating the update to the global Lyapunov behavior.
  3. [§3.1, §3.2, Tables 2 and 4] The comparison between Dynamic RL and conventional BPTT is confounded by differences in initialization. In Table 2 the actor's upper-hidden self-feedback spectral radius is 3.0 for Dynamic RL but 1.3 for conventional RL, and in Table 4 the actor output-layer initial weights are 0.1 for Dynamic RL and 0.0 for conventional RL. Since the claims of comparable success rates and faster adaptation in Figs. 9, 10, 15, and 16 depend directly on this comparison, the authors should either match these initialization choices across methods or systematically vary them to show that the observed differences are not artifacts of the initial dynamics. The author's own statement in Section 4.1 that parameter adjustment was not systematic reinforces this concern.
minor comments (5)
  1. [§2, Eq. (7)] Eq. (7) contains an apparent typo: the right-hand side uses eta_SRL where the learning rate for SAL, eta_SAL, is intended; this should be corrected for consistency with Eq. (8) and the surrounding text.
  2. [§3.2 vs Tables 2 and 3] The text after Fig. 16 says the truncated BPTT steps were '10 in the sequential navigation task and 5 in this task,' but Table 2 reports 20 for the navigation task and Table 3 reports 10 for the slider-crank task; the numbers in the text and tables need to be reconciled.
  3. [Appendix C, Eq. (24)] The 'exploration exponent' is a finite-time average of log distance increments with reset thresholds, not a standard Lyapunov exponent. The description should state this explicitly and specify how the distance is reinitialized after a reset, how episodes with early termination are handled, and why the state space is restricted to the input and upper-hidden dimensions.
  4. [Figs. 9 and 15] Success counts over 40 runs are reported without confidence intervals or statistical tests; for example, 36-2-2 versus 36-4-0 is not obviously significant, and statements about the 'highest success rate' should be supported by an appropriate test or interval.
  5. [§3.1, Fig. 11] The comparison of test steps between Dynamic RL and conventional RL is inherently uneven because in conventional RL exploration noise is removed during testing while in Dynamic RL exploration cannot be separated from the output; the paper acknowledges this in Section 4.1, but the point should be stated at the first presentation of Fig. 11 to prevent overinterpretation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Dynamic RL's SRL/SAL updates are definitions plus local gradients, and the global-control claims are validated by independent BPTT comparisons, ablations, and a measured exploration exponent.

full rationale

The derivation chain is not circular. The sensitivity s(U;w)=f'(U)||w|| (Eq. 4) is an explicit definition, and the SAL/SRL updates (Eqs. 7-17) are the corresponding local gradients of that sensitivity modulated by the TD error; this is algorithm construction, not a prediction of an independently defined target. The paper's substantive claims—that these updates control global convergence/divergence and improve learning—are tested outside the definition: against conventional BPTT actor-critic (Eqs. 18-19) on two tasks, through ablations (A, A-1, A-2 in Table 1), and with an independently computed exploration exponent (Appendix C) that is a measured Lyapunov-type quantity of the closed-loop system. The static-to-dynamic extension of Eq. (4) to the dynamic neuron of Eq. (1) is an explicit modeling assumption ('assuming that the infinitesimal variation dx of the input x changes slowly enough compared to the time constant tau'), not a circular reduction; its failure would be a correctness risk, not circularity. Self-citations—Shibata et al. 2021 for SAL and Shibata & Sakashita 2015 for chaos-based exploration—are not load-bearing: SAL is re-derived in Eqs. (7)-(10), and the chaotic-exploration claim is re-examined here through the measured exploration exponent and behavioral trajectories. The paper also acknowledges limitations (instability, output saturation, critic not yet dynamic, Sec. 4.1), which further indicates the results are presented as empirical findings rather than as consequences of the definitions. No fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central mechanism rests on three unproved but testable premises: sensitivity remains meaningful for dynamic recurrent neurons, local sensitivity updates can shape global closed-loop dynamics, and the TD-error sign is a sufficient controller for convergence and divergence. Most numerical constants are hand-set per task. No new physical entities are postulated.

free parameters (6)
  • Target sensitivity threshold sth = 1.3 (navigation), 1.6 (slider-crank)
    Hand-set values that switch each neuron between SAL and SRL in Fig. 4; the paper acknowledges sensitivity targets are task-dependent and not systematically optimized.
  • Initial spectral radius of actor upper-hidden self-feedback weights = 3.0 (Dynamic RL), 1.3 (conventional RL)
    Chosen to make the actor-environment loop chaotic for Dynamic RL; this design choice is load-bearing for deterministic exploration.
  • Per-layer learning rates = Table 4 values, e.g., eta_SRL 0.01 in->hid for navigation, 0.002 hid2->hid2 for slider-crank
    Hand-tuned separately for each task and layer; no sensitivity analysis is given.
  • Regularization rate eta_reg = 1e-6 (navigation), 1e-7 (slider-crank)
    Applied to output-layer weights to avoid tanh saturation; set by hand.
  • Critic raising threshold C_th and rate eta_raise = C_th=0.1; eta_raise=0.0005 (navigation), 0.0002 (slider-crank)
    A heuristic added to prevent the critic gradient from vanishing; values hand-set.
  • Moving average constants alpha and beta = alpha=0.001, beta=0.0001
    Time constants for sensitivity and critic moving averages; arbitrary small values chosen by hand.
assumptions (5)
  • domain assumption For a dynamic neuron obeying Eq. (1), the static sensitivity definition s(U;w)=f'(U)||w|| remains valid when input variations are slow compared to tau.
    Invoked immediately after Eq. (4) in Section 2; it is not proven for recurrent inputs arriving at t-1.
  • domain assumption Large random initial weights with spectral radius 3.0, maintained by SAL, produce chaotic agent-environment dynamics with a positive exploration exponent.
    Used in Section 3.1 and Table 2 to justify deterministic exploration; validated only empirically through the exploration exponent in Fig. 13.
  • domain assumption Reducing neuron sensitivity around positive-TD transitions increases reproducibility, and increasing it around negative-TD transitions increases exploration, in the full closed-loop system.
    This is the learning hypothesis in Fig. 5 and Section 2; the paper verifies it behaviorally but provides no proof that local sensitivity controls the relevant Lyapunov directions.
  • domain assumption The TD error r_hat = gamma C_{t+1} + r_{t+1} - C_t is a sufficient scalar signal for adjusting dynamics.
    Standard actor-critic assumption inherited by the method; not an ad hoc invention.
  • domain assumption The finite-time, single-perturbation exploration exponent in Appendix C captures the relevant chaoticity of the closed-loop system.
    Appendix C; the exponent is a finite-time, single-trajectory Lyapunov-like estimate, not a rigorous Lyapunov spectrum.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Reinforcement Learning for Actors." pith.science (2026). https://pith.science/paper/IZRA3KFQ

@misc{pith2026250210200,
  author       = {Pith},
  title        = {Pith review of: Dynamic Reinforcement Learning for Actors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IZRA3KFQ}},
  note         = {Machine review of arXiv:2502.10200}
}
read the original abstract

Dynamic Reinforcement Learning (Dynamic RL), proposed in this paper, directly controls system dynamics, instead of the actor (action-generating neural network) outputs at each moment, bringing about a major qualitative shift in reinforcement learning (RL) from static to dynamic. The actor is initially designed to generate chaotic dynamics through the loop with its environment, enabling the agent to perform flexible and deterministic exploration. Dynamic RL controls global system dynamics using a local index called "sensitivity," which indicates how much the input neighborhood contracts or expands into the corresponding output neighborhood through each neuron's processing. While sensitivity adjustment learning (SAL) prevents excessive convergence of the dynamics, sensitivity-controlled reinforcement learning (SRL) adjusts them -- to converge more to improve reproducibility around better state transitions with positive TD error and to diverge more to enhance exploration around worse transitions with negative TD error. Dynamic RL was applied only to the actor in an Actor-Critic RL architecture while applying it to the critic remains a challenge. It was tested on two dynamic tasks and functioned effectively without external exploration noise or backward computation through time. Moreover, it exhibited excellent adaptability to new environments, although some problems remain. Drawing parallels between 'exploration' and 'thinking,' the author hypothesizes that "exploration grows into thinking through learning" and believes this RL could be a key technique for the emergence of thinking, including inspiration that cannot be reconstructed from massive existing text data. Finally, despite being presumptuous, the author presents the argument that this research should not proceed due to its potentially fatal risks, aiming to encourage discussion.

Figures

Figures reproduced from arXiv: 2502.10200 by the authors.

Figure 1
Figure 1. The difference in exploration between conventional RL and humans, who have inspired the exploration in the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The author’s concept of the relationship between ‘exploration’ and ‘thinking’ and how they relate to system [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. A conceptual diagram explaining the degrees of freedom (DOFs) that dynamics have for a sample case of [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Dynamic RL applies either SAL or SRL depending on the condition in each neuron. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Basic concept of Dynamic RL (or more specifically, SRL) proposed in this paper. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: A conceptual diagram of conventional RL. RL aims to control the actor output vector based on the TD error. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: A conceptual diagram of Dynamic Reinforcement Learning (RL). RL aims to control the convergence or [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Sequential navigation task and the RNN used in the simulation. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Comparison of success rate among six learning conditions as in Table 1 in the sequential navigation task. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Learning curve for the sequential navigation task. The vertical axis indicates the average number of [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Comparison of the average number of steps in the final stage of learning and in the test among four learning [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Comparison of sample agent behaviors at four stages in learning between (A) Dynamic RL and (B-1) [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Comparison of changes in exploration exponents in the sequential navigation task among four learning con [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Slider-crank control task as a dynamic pattern generation task and the RNN used. The agent applies a force [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Comparison of success rate among six learning conditions in Table 1 in the slider-crank control task. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Comparison of changes in the average number of revolutions per episode, calculated over every 100 [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Comparison of changes in (a) rotor angle with four sensor signals and a reinforcement signal (reward or [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Changes in (a) rotor angle with four sensor signals and a reinforcement signal (reward or penalty), (b) actor [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: A conceptual diagram explaining how Dynamic RL works in simplified 2-D state space at the expense of [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: ‘Exploration’ and ‘thinking’ loop and its relationship to Dynamic RL, the author advocates. This is an [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 45 canonical work pages

  1. [1]

    Can I say, now machines can think?

    N. Aggarwal, G. J. Saxena, S. Singh, and A. Pundir. Can I say, now machines can think?, 2023. URL https://arxiv.org/abs/2307.07526

    work page Pith review arXiv 2023

  2. [2]

    Aihara, T

    K. Aihara, T. Takabe, and M. Toyoda. Chaotic neural networks. Physics Letters A, 144 0 (6): 0 333--340, 1990. ISSN 0375-9601. doi:https://doi.org/10.1016/0375-9601(90)90136-C. URL https://www.sciencedirect.com/science/article/pii/037596019090136C

    arXiv 1990

  3. [3]

    S. Amari. Any target function exists in a neighborhood of any sufficiently wide random network: A geometrical perspective, 2020. URL https://arxiv.org/abs/2001.06931

    work page Pith review arXiv 2020

  4. [4]

    Andrychowicz, F

    M. Andrychowicz, F. Wolski, A. Ray, et al. Hindsight experience replay. In I. Guyon et al., editors, Advances in Neural Information Processing Systems, volume 30, pages 1--11. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/453fadbd8a1a3af50a9df4df899537b5-Paper.pdf

    work page 2017

  5. [5]

    Azizi and G

    T. Azizi and G. Kerr. Application of stability theory in study of local dynamics of nonlinear systems. Journal of Applied Mathematics and Physic, 8: 0 1180--1192, 2020. doi:https://doi.org/10.4236/jamp.2020.86089

    work page arXiv 2020

  6. [6]

    Berlyne and W

    D. Berlyne and W. E. Vinacke. Thought (n.d.), 2025. URL https://www.britannica.com/topic/thought. In Encyclopaedia Britannica, Encyclopedia Britannica Inc. Retrieved January 11, 2025

    work page 2025

  7. [7]

    Broad, S

    T. Broad, S. Berns, S. Colton, and M. Grierson. Active divergence with generative deep learning - a survey and taxonomy, 2021. URL https://arxiv.org/abs/2107.05599

    arXiv 2021

  8. [8]

    Statement on AI risk, 2025 a

    Center for AI Safety . Statement on AI risk, 2025 a . URL https://www.safe.ai/work/statement-on-ai-risk/

    work page 2025

Show all 69 references
  1. [9]

    An overview of catastrophic AI risks, 2025 b

    Center for AI Safety . An overview of catastrophic AI risks, 2025 b . URL https://www.safe.ai/ai-risk

    2025

  2. [10]

    Chakrabarty, P

    T. Chakrabarty, P. Laban, D. Agarwal, et al. Art or artifice? large language models and the false promise of creativity, 2024. URL https://arxiv.org/abs/2309.14556

    2024 arXiv

  3. [11]

    The alternative uses test, 2018

    CreativeHuddle. The alternative uses test, 2018. URL https://www.creativehuddle.co.uk/post/the-alternative-uses-test

    2018

  4. [12]

    E. de Bono. The Use of Lateral Thinking. Avon Books, 1971

    1971

  5. [13]

    K. Doya, A. Ema, H. Kitano, et al. Social impact and governance of AI and neurotechnologies. Neural Networks, 152: 0 542--554, 2022. ISSN 0893-6080. doi:https://doi.org/10.1016/j.neunet.2022.05.012. URL https://www.sciencedirect.com/science/article/pii/S0893608022001861

    2022 doi

  6. [14]

    Franceschelli and M

    G. Franceschelli and M. Musolesi. Creative beam search: Llm-as-a-judge for improving response generation, 2024. URL https://arxiv.org/abs/2405.00099v4

    2024 arXiv

  7. [15]

    W. J. Freeman. The physiology of perception. Scientific American, 264 0 (2): 0 78--85, February 1991

    1991

  8. [16]

    Fujimoto, H

    S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the International Conference on Machine Learning (ICML), pages 1587--1596, 2018

    2018

  9. [17]

    Research priorities for robust and beneficial artificial intelligence: An open letter, 2015

    Future of Life Institute . Research priorities for robust and beneficial artificial intelligence: An open letter, 2015. URL https://futureoflife.org/open-letter/ai-open-letter/

    2015

  10. [18]

    Gendron, Q

    G. Gendron, Q. Bao, M. Witbrock, and G. Dobbie. Large language models are not strong abstract reasoners, 2024. URL https://arxiv.org/abs/2305.19555

    2024 arXiv

  11. [19]

    Goto and K

    K. Goto and K. Shibata. Acquisition of deterministic exploration and purposive memory through reinforcement learning with a recurrent neural network. In Proc. of SICE Annual Conf. 2010, pages FB03--1, 2010

    2010

  12. [20]

    J. P. Guilford. The nature of human intelligence. McGraw-Hill, 1967

    1967

  13. [21]

    E. E. Guzik, C. Byrge, and C. Gilde. The originality of machines: AI takes the torrance test. Journal of Creativity, 33 0 (3): 0 100065, 2023. ISSN 2713-3745. doi:https://doi.org/10.1016/j.yjoc.2023.100065. URL https://www.sciencedirect.com/science/article/pii/S2713374523000249

    2023

  14. [22]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning (ICML), pages 1861--1870, 2018

    2018

  15. [23]

    Huang, S

    S. Huang, S. Ma, Y. Li, et al. Lateval: An interactive llms evaluation benchmark with incomplete information from lateral thinking puzzles. In Proc. of LREC-COLING 2024 Torino, pages 10186--10197, 2024

    2024

  16. [24]

    Ismayilzada, D

    M. Ismayilzada, D. Paul, A. Bosselut, and L. van der Plas. Creativity in AI : Progresses and challenges, 2024. URL https://arxiv.org/abs/2410.17218

    2024 arXiv

  17. [25]

    Jiang, F

    Y. Jiang, F. Ilievski, K. Ma, and Z. Sourati. Brainteaser: Lateral thinking puzzles for large language models, 2023. URL https://arxiv.org/abs/2310.05057

    2023 arXiv

  18. [26]

    Creativity

    J.P.Guilford. Creativity. The American Psychologist, 13: 0 444--454, 1950

    1950

  19. [27]

    Khachaturyan, S

    A. Khachaturyan, S. Semenovskaya, and B. Vainshtein. The thermodynamic approach to the structure analysis of crystals. Acta Cryst., A37: 0 742--754, 1981

    1981

  20. [28]

    Koivisto and S

    M. Koivisto and S. Grassini. Best humans still outperform artificial intelligence in a creative divergent thinking task. Scientific Report, 13 0 (13601), 2023

    2023

  21. [29]

    T. H. Kung, M. Cheatham, A. Medenilla, et al. Performance of ChatGPT on USMLE : Potential for AI -assisted medical education using large language models. PLOS Digital Health, 2 0 (2): 0 e0000198, 2023

    2023

  22. [30]

    Kurzweil

    R. Kurzweil. The age of spiritual machines: when computers exceed human intelligence. Viking Press, USA, 1998. ISBN 0670882178

    1998

  23. [31]

    X. Lu, M. Sclar, S. Hallinan, et al. AI as humanity's Salieri : Quantifying linguistic creativity of language models via systematic attribution of machine text against web text, 2025. URL https://arxiv.org/abs/2410.04265

    2025 arXiv

  24. [32]

    Matsuki and K

    T. Matsuki and K. Shibata. Adaptive balancing of exploration and exploitation around the edge of chaos in internal-chaos-based learning. Neural Networks, 132: 0 19--29, 2020

    2020

  25. [33]

    Matsuki, Y

    T. Matsuki, Y. Sakemi, and K. Aihara. Chaos-based reinforcement learning with TD3 , 2024. URL https://arxiv.org/abs/2405.09086

    2024

  26. [34]

    McCulloch and W

    W. McCulloch and W. Pitts. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5: 0 115--133, 1943

    1943

  27. [35]

    Mitchell, A

    M. Mitchell, A. B. Palmarini, and A. Moskvichev. Comparing humans, GPT-4 , and GPT-4V on abstraction and reasoning tasks, 2023. URL https://arxiv.org/abs/2311.09247

    2023 arXiv

  28. [36]

    V. Mnih, K. Kavukcuoglu, D. Silver, et al. Human-level control through deep reinforcement learning. Nature, 518 0 (7540): 0 529--533, 2015. doi:10.1038/nature14236. URL https://doi.org/10.1038/nature14236

    2015 doi

  29. [37]

    V. Mnih, A. P. Badia, M. Mirza, et al. Asynchronous methods for deep reinforcement learning, 2016. URL https://arxiv.org/abs/1602.01783

    2016 arXiv

  30. [38]

    S. S. Nath, P. Dayan, and C. Stevenson. Characterising the creative process in humans and large language models, 2024. URL https://arxiv.org/abs/2405.00899

    2024 arXiv

  31. [39]

    Newell, J

    A. Newell, J. C. Shaw, and H. A. Simon. The processes of creative thinking. In Symposium on Creative Thinking, pages 1--82 (P--1320), 1959

    1959

  32. [40]

    OpenAI Five , 2019

    OpenAI . OpenAI Five , 2019. URL https://openai.com/index/openai-five-defeats-dota-2-world-champions/. Accessed: 2024-08-24

    2019

  33. [41]

    GPT-4 , 2023

    OpenAI. GPT-4 , 2023. URL https://openai.com/index/gpt-4-research/. Accessed: 2025-1-26

    2023

  34. [42]

    Achiam, et al

    OpenAI , J. Achiam, et al. GPT-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  35. [43]

    Peeperkorn, T

    M. Peeperkorn, T. Kouwenhoven, D. Brown, and A. Jordanous. Is temperature the creativity parameter of large language models?, 2024. URL https://arxiv.org/abs/2405.00492

    2024 arXiv

  36. [44]

    Pichai, D

    S. Pichai, D. Hassabis, and K. Kavukcuoglu. Gemini: An adaptive multi-modal AI system, 2024. URL https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/. Accessed: 2025-1-26

    2024

  37. [45]

    M. E. Raichle, A. M. MacLeod, A. Z. Snyder, et al. A default mode of brain function. Proceedings of the National Academy of Sciences - PNAS, 98 0 (2): 0 676--682, 2001. ISSN 0027-8424

    2001

  38. [46]

    D. E. Rumelhart and J. L. McClelland. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1: Foundations. MIT Press, 1986

    1986

  39. [47]

    Sawatsubashi, M

    Y. Sawatsubashi, M. F. bin Samsudin, and K. Shibata. Emergence of discrete and abstract state representation through reinforcement learning in a continuous input task. In Advances in Intelligent Systems and Computing, Robot Intelligence Technology and Applications 2012, pages ...

    2012

  40. [48]

    Schaul, J

    T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay, 2016. URL https://arxiv.org/abs/1511.05952

    2016 arXiv

  41. [49]

    Schrittwieser, I

    J. Schrittwieser, I. Antonoglou, T. Hubert, et al. Mastering Atari, Go , chess and shogi by planning with a learned model. Nature, 588 0 (7839): 0 604--609, Dec. 2020. ISSN 1476-4687. doi:10.1038/s41586-020-03051-4. URL http://dx.doi.org/10.1038/s41586-020-03051-4

    2020 doi

  42. [50]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms. In Proceedings of the International Conference on Machine Learning (ICML), pages 1--9, 2017

    2017

  43. [51]

    K. Shibata. Learning of deterministic exploration and temporal abstraction in reinforcement learning. In Proc. of SICE-ICCAS (SICE-ICASE Int'l Joint Conf.), pages 4569--4574, 2006

    2006

  44. [52]

    K. Shibata. Emergence of intelligence through reinforcement learning with a neural network. In A. Mellouk, editor, Advances in Reinforcement Learning, chapter 6, pages 99--120. IntechOpen, 2011. doi:10.5772/13443. URL https://doi.org/10.5772/13443

    2011 doi

  45. [53]

    K. Shibata. Communications that emerge through reinforcement learning using a (recurrent) neural network, 2017 a . URL https://arxiv.org/abs/1703.03543

    2017 arXiv

  46. [54]

    K. Shibata. Functions that emerge through end-to-end reinforcement learning - the direction for artificial general intelligence -, 2017 b . URL https://arxiv.org/abs/1703.02239

    2017 arXiv

  47. [55]

    Shibata and K

    K. Shibata and K. Ito. Emergence of communication for negotiation by a recurrent neural network. In Proc. of ISADS (Int'l Symposium on Autonomous Decentralized System) '99, pages 294--301, 1999

    1999

  48. [56]

    Shibata and Y

    K. Shibata and Y. Okabe. Reinforcement learning when visual sensory signals are directly given as inputs. In Proc. of ICNN (Int'l Conf. on Neural Networks) '97 Houston, volume 3, pages 1716--1720, 1997

    1997

  49. [57]

    Shibata and Y

    K. Shibata and Y. Sakashita. Reinforcement learning with internal-dynamics-based exploration using a chaotic neural network. In Proc. of Int'l Joint Conf. on Neural Networks (IJCNN) 2015, page \#15231, 2015

    2015

  50. [58]

    Shibata, T

    K. Shibata, T. Ejima, Y. Tokumaru, and T. Matsuki. Sensitivity -- local index to control chaoticity or gradient globally --. Neural Networks, 143: 0 436--451, 2021. ISSN 0893-6080

    2021

  51. [59]

    C. A. Skarda and W. J. Freeman. How brains make chaos in order to make sense of the world. Behavioral and brain sciences, 10: 0 161--173, 1987

    1987

  52. [60]

    R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018

    2018

  53. [61]

    D. Tam, A. Mascarenhas, S. Zhang, et al. Evaluating the factual consistency of large language models through news summarization, 2023. URL https://arxiv.org/abs/2211.08412

    2023 arXiv

  54. [62]

    E. P. Torrance. Torrance tests of creative thinking, 2018. URL https://www.ststesting.com/gift/TTCT_InterpMOD.2018.pdf

    2018

  55. [63]

    A. M. Turing. Computing machinery and intelligence. Mind, New Series, 59 0 (236): 0 433--460, 1950. URL https://www.jstor.org/stable/2251299

    1950

  56. [64]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, et al. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  57. [65]

    Vinyals, I

    O. Vinyals, I. Babuschkin, W. Czarnecki, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575: 0 350--354, 2019. doi:10.1038/s41586-019-1724-z. URL https://doi.org/10.1038/s41586-019-1724-z

    2019 doi

  58. [66]

    R. Woolf. Plato and the norms of thought. Mind, 122: 0 171--216, 2013. URL https://doi.org/10.1093/mind/fzt012

    2013 doi

  59. [67]

    Yamashita and J

    Y. Yamashita and J. Tani. Emergence of Functional Hierarchy in a Multiple Timescale Neural Network Model: A Humanoid Robot Experiment . PLOS Computational Biology, 6 0 (10), 2010. doi:10.1371/journal.pcbi.1000220

    2010 doi

  60. [68]

    Yudkowsky

    E. Yudkowsky. Pausing AI developments isn't enough. we need to shut it all down. Time Magazine, 2023. URL https://time.com/6266923/ai-eliezer-yudkowsky-open-letter\ -not-enough/

    2023

  61. [69]

    Y. Zhao, R. Zhang, W. Li, et al. Assessing and understanding creativity in large language models, 2024. URL https://arxiv.org/pdf/2401.12491v1

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.