pith. sign in

arxiv: 2606.21350 · v1 · pith:WSWKWZAEnew · submitted 2026-06-19 · 💻 cs.LG

A Reward-Petri-Net Interpretation of Temporal Behavior Trees

Till Schmeil , G\"unther Waxenegger-Wilfing , Sebastian Schirmer This is my paper

Pith reviewed 2026-06-26 14:19 UTC · model grok-4.3

classification 💻 cs.LG
keywords Temporal Behavior TreesReward-Petri-NetsReinforcement LearningReward DesignPetri NetsLinear Temporal LogicRobotic TasksHierarchical Control
0
0 comments X

The pith

Temporal Behavior Trees translate into Reward-Petri-Nets that generate structure-based rewards for reinforcement learning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes a direct translation from Temporal Behavior Trees, which extend standard behavior trees by placing Linear Temporal Logic constraints in their leaf nodes, into Petri Nets. Rewards are then assigned automatically according to the resulting net structure rather than through manual design. Experiments in robotic environments of rising complexity show these rewards let reinforcement learning succeed where standard methods fail and raise sample efficiency. A reader would care because reward specification remains a central obstacle when applying RL to tasks that combine hierarchical structure with explicit timing requirements.

Core claim

Temporal Behavior Trees with Linear Temporal Logic leaves can be converted into Reward-Petri-Nets by a structure-preserving translation to a Petri Net followed by automatic reward placement derived from the net's places, transitions, and operator semantics, so that the resulting rewards guide policy search in long-horizon tasks.

What carries the argument

The translation from a Temporal Behavior Tree to a Petri Net with rewards assigned from the net's structural elements.

If this is right

  • TBT-derived rewards allow successful learning in environments where vanilla RL produces no progress.
  • Sample efficiency rises relative to hand-crafted reward baselines across multiple task difficulties.
  • Different choices of reward distribution within the same RPN structure give direct control over learning speed and order.
  • The same TBT can be used both to specify the task and to generate the rewards that train the policy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Petri Net representation could support formal verification of the generated reward function before training begins.
  • The approach might reduce the need for environment-specific reward tuning when moving between related robotic tasks.
  • Existing Petri Net analysis tools could be applied to detect unreachable states or conflicting constraints in the original TBT.

Load-bearing premise

The translation from Temporal Behavior Trees to Petri Nets preserves every temporal constraint and behavioral rule without loss or alteration.

What would settle it

A policy trained with the RPN rewards that violates one of the original Linear Temporal Logic leaf constraints or fails to complete the task sequence encoded by the TBT operators.

Figures

Figures reproduced from arXiv: 2606.21350 by G\"unther Waxenegger-Wilfing, Sebastian Schirmer, Till Schmeil.

Figure 1
Figure 1. Figure 1: illustrates an example TBT specification. The root is a parallel node, annotated with the subscript (1), indicating that satisfaction of at least one of its subtrees is sufficient. The left subtree consists of the temporal logic formula a, which specifies that proposition a must eventually hold. The right subtree is a sequence node that evaluates its children from left to right: first, the agent must event… view at source ↗
Figure 2
Figure 2. Figure 2: Deterministic Petri net 4 Building Reward Petri Nets In this section, we extend Petri nets to Reward Petri nets (RPN), present the construction of (RPN) from TBT specifi￾cation, and show how rewards are assigned given distribution functions. RPNs extend Petri nets by outputting rewards and actions upon firing transitions and adding guards to transitions. Definition 6 (Reward Petri Net). A Reward Petri net … view at source ↗
Figure 3
Figure 3. Figure 3: The colored regions show the mapping between the TBT [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Excerpt of a MiniGrid-ObstructedMaze-1Q gen￾eration where the agent’s (red triangle) FOV is depicted in light grey. We apply the PPO algorithm introduced by [Schulman et al., 2017] as implemented by Stable Baselines3 [Raffin et al., 2021] in addition to the related RL Baselines3 Zoo [Raffin, 2020]. Hyperparameters are included in the supple￾mentary material. For comparability, functionally identical SB3–Mi… view at source ↗
Figure 5
Figure 5. Figure 5: TBT for Unlock environment. Consult [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: TBT for UnlockPickup environment. Nodes adopted from Tu shown in gray (compare with [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: TBT for BlockedUnlockPickup environment. Nodes adopted from Tup shown in gray (compare with [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: TBT for ObstructedMaze-2Dlh environment. Uses T ′ u (see [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: TBT for ObstructedMaze-1Q environment. Uses T ′′ u (see [PITH_FULL_IMAGE:figures/full_fig_p006_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Comparison of training performance on a series of increasingly hard MiniGrid environments. [PITH_FULL_IMAGE:figures/full_fig_p007_10.png] view at source ↗
read the original abstract

This paper introduces an interpretation of Temporal Behavior Trees (TBTs) as Reward-Petri-Nets (RPNs) for reinforcement learning (RL). Designing reward functions for complex, long-horizon robotic tasks is notoriously difficult, especially when tasks have hierarchical structure and temporal constraints. TBTs extend conventional behavior trees (BTs) used in robotic applications by incorporating temporal properties into their leaf nodes. This allows TBTs to represents not only the behavioral task structure defined by BT operators such as Sequence, Fallback, and Parallel, but also the task's temporal constraints. In this work, the constraints are specified in the leaf nodes using Linear Temporal Logic. In order to inform RL rewards using TBTs, we provide a translation from TBT into a Petri Net (PN) and show how rewards can be automatically assigned based on the TBT's structure, resulting in a RPN. In a series of increasingly challenging environments, we demonstrate how TBT-based rewards enable learning where vanilla RL fails, improve sample efficiency, and offer flexible, intuitive control over the learning progress. We showcase the learning impact by using different reward distribution schemes and TBT structures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims to interpret Temporal Behavior Trees (TBTs) with LTL-specified leaf nodes as Reward-Petri-Nets (RPNs) via a structure-preserving translation from TBT to Petri Net followed by automatic reward assignment based on the TBT structure. It reports experimental results in a series of increasingly challenging environments showing that the resulting RPN rewards enable successful policy learning where vanilla RL fails, improve sample efficiency, and allow flexible control over learning via different reward distribution schemes and TBT structures.

Significance. If the translation is shown to preserve all temporal semantics of the TBT (including LTL constraints) and the structure-derived rewards prove sufficient without post-hoc tuning, the work would provide a useful bridge between hierarchical task specifications used in robotics and reward design for RL, potentially reducing manual reward engineering for long-horizon tasks.

major comments (2)
  1. [Abstract] Abstract: the central claim that 'rewards can be automatically assigned based on the TBT's structure' and suffice to guide learning rests on the unverified assumption that the TBT-to-PN translation preserves temporal semantics; no formal equivalence or preservation argument is referenced, making it impossible to assess whether the RPN rewards are faithful or merely heuristic.
  2. [Abstract] Abstract (experimental claim): the statement that TBT-based rewards 'enable learning where vanilla RL fails' and 'improve sample efficiency' cannot be evaluated without details on environment definitions, baseline implementations, number of runs, or whether reward schemes involved environment-specific adjustments; the reported success may depend on post-hoc choices not disclosed in the high-level description.
minor comments (1)
  1. [Abstract] The abstract introduces the term 'Reward-Petri-Net (RPN)' without a concise definition or pointer to the section where the construction is formalized.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their comments. We address each major comment below with references to the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that 'rewards can be automatically assigned based on the TBT's structure' and suffice to guide learning rests on the unverified assumption that the TBT-to-PN translation preserves temporal semantics; no formal equivalence or preservation argument is referenced, making it impossible to assess whether the RPN rewards are faithful or merely heuristic.

    Authors: Section 3 defines the TBT-to-PN translation operator-by-operator, mapping Sequence, Fallback, Parallel, and LTL leaf nodes to PN places and transitions such that execution traces in the PN correspond to TBT satisfaction. Rewards are assigned directly from this structure (Section 4). The construction is semantics-preserving by design, though the manuscript does not include a formal bisimulation proof. We will revise the abstract to reference Section 3. revision: partial

  2. Referee: [Abstract] Abstract (experimental claim): the statement that TBT-based rewards 'enable learning where vanilla RL fails' and 'improve sample efficiency' cannot be evaluated without details on environment definitions, baseline implementations, number of runs, or whether reward schemes involved environment-specific adjustments; the reported success may depend on post-hoc choices not disclosed in the high-level description.

    Authors: The abstract summarizes results at a high level. Full details appear in Section 5.1 (environments), Section 5.2 (baselines), Section 5 (10 independent runs per experiment), and Section 4.3 (reward schemes). Rewards are assigned automatically from the TBT structure with no environment-specific post-hoc tuning. No revision to the abstract is needed. revision: no

Circularity Check

0 steps flagged

No significant circularity; translation is constructive and self-contained

full rationale

The paper's core step is a structural translation from TBT (with LTL leaves) to Petri Net followed by automatic reward placement derived from the tree operators and structure. This is presented as an explicit mapping and reward scheme definition, not a derivation that reduces by construction to fitted parameters or prior self-citations. No equations or steps are described that equate a 'prediction' to an input fit, import uniqueness from the authors' own prior work as an external theorem, or rename known results. Experiments are framed as empirical demonstrations of the defined method in environments, not as forced outcomes. The derivation chain is therefore independent of its own outputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Central claim rests on the correctness of the TBT-to-PN translation and the assumption that structure-derived rewards suffice for RL; no free parameters are mentioned in the abstract.

axioms (1)
  • domain assumption Linear Temporal Logic semantics correctly capture the temporal constraints placed in TBT leaf nodes
    Invoked when the paper states that TBTs represent temporal constraints via LTL in the leaves.
invented entities (1)
  • Reward-Petri-Net (RPN) no independent evidence
    purpose: Petri Net obtained from TBT that carries automatically assigned rewards for RL
    New composite object introduced by the translation; no independent evidence supplied in abstract.

pith-pipeline@v0.9.1-grok · 5742 in / 1280 out tokens · 26252 ms · 2026-06-26T14:19:54.639725+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 1 canonical work pages

  1. [1]

    Rudder: Return decomposition for delayed rewards.Advances in Neural Information Processing Systems, 32,

    [Arjona-Medinaet al., 2019 ] Jose Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Jo- hannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards.Advances in Neural Information Processing Systems, 32,

    2019

  2. [2]

    The option-critic architecture

    [Baconet al., 2017 ] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. InProceed- ings of the AAAI conference on artificial intelligence, vol- ume 31,

    2017

  3. [3]

    Aguilar, Dejan Ni ˇckovi´c, and Jyotir- moy V

    [Balakrishnanet al., 2023 ] Anand Balakrishnan, Stefan Jakˇsi´c, Edgar A. Aguilar, Dejan Ni ˇckovi´c, and Jyotir- moy V . Deshmukh. Model-free reinforcement learning for spatiotemporal tasks using symbolic automata. In2023 62nd IEEE Conference on Decision and Control (CDC), pages 6834–6840,

    2023

  4. [4]

    Runtime verification for ltl and tltl.ACM Trans

    [Baueret al., 2011 ] Andreas Bauer, Martin Leucker, and Christian Schallhart. Runtime verification for ltl and tltl.ACM Trans. Softw. Eng. Methodol., 20(4), September

    2011

  5. [5]

    Unifying count-based exploration and intrinsic motivation.Advances in neural information processing systems, 29,

    [Bellemareet al., 2016 ] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation.Advances in neural information processing systems, 29,

    2016

  6. [6]

    Random search for hyper-parameter optimization

    [Bergstra and Bengio, 2012] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281–305, February

    2012

  7. [7]

    Klassen, Richard Valenzano, and Sheila A

    [Camachoet al., 2019 ] Alberto Camacho, Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, and Sheila A. McIlraith. Ltl and beyond: Formal languages for reward function specification in reinforcement learning. InProceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 6065–6073. International Joint...

    2019

  8. [8]

    Reward machines for vision-based robotic ma- nipulation

    [Camachoet al., 2021 ] Alberto Camacho, Jacob Varley, Andy Zeng, Deepali Jain, Atil Iscen, and Dmitry Kalash- nikov. Reward machines for vision-based robotic ma- nipulation. In2021 IEEE International Conference on Robotics and Automation (ICRA), pages 14284–14290,

    2021

  9. [9]

    Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks,

    [Chevalier-Boisvertet al., 2023 ] Maxime Chevalier- Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks.CoRR, abs/2306.13831,

    work page arXiv 2023

  10. [10]

    Hyperparameters in reinforcement learning and how to tune them,

    [Eimeret al., 2023 ] Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in reinforcement learning and how to tune them,

    2023

  11. [11]

    Ltlf2dfa, March

    [Fuggitti, 2019] Francesco Fuggitti. Ltlf2dfa, March

    2019

  12. [12]

    Deep reinforce- ment learning with temporal logics

    [Hasanbeiget al., 2020 ] Mohammadhosein Hasanbeig, Daniel Kroening, and Alessandro Abate. Deep reinforce- ment learning with temporal logics. In Nathalie Bertrand and Nils Jansen, editors,Formal Modeling and Analysis of Timed Systems, pages 1–22, Cham,

    2020

  13. [13]

    [Jaderberget al., 2017 ] Max Jaderberg, V olodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu

    Springer International Publishing. [Jaderberget al., 2017 ] Max Jaderberg, V olodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. InInterna- tional Conference on Learning Representations,

    2017

  14. [14]

    Prioritized level replay

    [Jianget al., 2021 ] Minqi Jiang, Edward Grefenstette, and Tim Rockt¨aschel. Prioritized level replay. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th Interna- tional Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4940–

    2021

  15. [15]

    Sample-efficient reinforcement learning with temporal logic objectives: Leveraging the task specification to guide exploration,

    [Kantaros and Wang, 2024] Yiannis Kantaros and Jun Wang. Sample-efficient reinforcement learning with temporal logic objectives: Leveraging the task specification to guide exploration,

    2024

  16. [16]

    [Klissarov and Machado, 2023] Martin Klissarov and Mar- los C. Machado. Deep laplacian-based options for temporally-extended exploration,

    2023

  17. [17]

    Reinforcement learning with temporal logic re- wards,

    [Liet al., 2017 ] Xiao Li, Cristian-Ioan Vasile, and Calin Belta. Reinforcement learning with temporal logic re- wards,

    2017

  18. [18]

    Machado, Andre Barreto, Doina Precup, and Michael Bowling

    [Machadoet al., 2023 ] Marlos C. Machado, Andre Barreto, Doina Precup, and Michael Bowling. Temporal abstrac- tion in reinforcement learning with the successor repre- sentation,

    2023

  19. [19]

    Policy invariance under reward transformations: Theory and application to reward shaping

    [Nget al., 1999 ] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. InInternational conference on machine learning, volume 16,

    1999

  20. [20]

    Curiosity-driven exploration by self-supervised prediction

    [Pathaket al., 2017 ] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. InInternational conference on machine learning, pages 2778–2787. PMLR,

    2017

  21. [21]

    Kommunikation mit auto- maten

    [Petri, 1962] Carl Adam Petri. Kommunikation mit auto- maten

    1962

  22. [22]

    The temporal logic of programs

    [Pnueli, 1977] Amir Pnueli. The temporal logic of programs. In18th Annual Symposium on Foundations of Computer Science (sfcs 1977), pages 46–57,

    1977

  23. [23]

    Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8,

    [Raffinet al., 2021 ] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8,

    2021

  24. [24]

    Rl baselines3 zoo

    [Raffin, 2020] Antonin Raffin. Rl baselines3 zoo. https:// github.com/DLR-RM/rl-baselines3-zoo,

    2020

  25. [25]

    Lamaconv - logics and automata converter li- brary

    [Scheffel and Schmitz, 2016] Torben Scheffel and Malte Schmitz. Lamaconv - logics and automata converter li- brary. https://www.isp.uni-luebeck.de/lamaconv,

    2016

  26. [26]

    Temporal behavior trees: Robust- ness and segmentation

    [Schirmeret al., 2024 ] Sebastian Schirmer, Jasdeep Singh, Emily Jensen, Johann Dauer, Bernd Finkbeiner, and Sri- ram Sankaranarayanan. Temporal behavior trees: Robust- ness and segmentation. InProceedings of the 27th ACM International Conference on Hybrid Systems: Computa- tion and Control, HSCC ’24, New York, NY , USA,

    2024

  27. [27]

    [Schmidhuber, 1991] J¨urgen Schmidhuber

    Association for Computing Machinery. [Schmidhuber, 1991] J¨urgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. InProc. of the international confer- ence on simulation of adaptive behavior: From animals to animats, pages 222–227,

    1991

  28. [28]

    Proximal policy optimization algorithms,

    [Schulmanet al., 2017 ] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms,

    2017

  29. [29]

    Sutton and Andrew G

    [Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 2 edition,

    2018

  30. [30]

    Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning

    [Suttonet al., 1999 ] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181–211,

    1999

  31. [31]

    #explo- ration: A study of count-based exploration for deep rein- forcement learning.Advances in neural information pro- cessing systems, 30,

    [Tanget al., 2017 ] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. #explo- ration: A study of count-based exploration for deep rein- forcement learning.Advances in neural information pro- cessing systems, 30,

    2017

  32. [32]

    Re- ward machines: Exploiting reward function structure in reinforcement learning.Journal of Artificial Intelligence Research, 73:173–208,

    [Toro Icarteet al., 2022 ] Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A McIlraith. Re- ward machines: Exploiting reward function structure in reinforcement learning.Journal of Artificial Intelligence Research, 73:173–208,

    2022

  33. [33]

    mvcisback/py-metric-temporal-logic: v0.1.1, January

    [Vazquez-Chanlatte, 2019] Marcell Vazquez-Chanlatte. mvcisback/py-metric-temporal-logic: v0.1.1, January

    2019

  34. [34]

    Revisiting intrin- sic reward for exploration in procedurally generated envi- ronments

    [Wanget al., 2023 ] Kaixin Wang, Kuangqi Zhou, Bingyi Kang, Jiashi Feng, and Shuicheng Yan. Revisiting intrin- sic reward for exploration in procedurally generated envi- ronments. InThe Eleventh International Conference on Learning Representations,

    2023

  35. [35]

    Tractable reinforcement learning for signal temporal logic tasks with counterfactual experience replay

    [Wanget al., 2024 ] Siqi Wang, Xunyuan Yin, Shaoyuan Li, and Xiang Yin. Tractable reinforcement learning for signal temporal logic tasks with counterfactual experience replay. IEEE Control Systems Letters, 8:616–621,

    2024

  36. [36]

    pytransitions/transitions

    [Yarkoni and Neumann, 2025] Tal Yarkoni and Andreas Neumann. pytransitions/transitions. https://github.com/ pytransitions/transitions, 2025

    2025