Pith. sign in

REVIEW 4 major objections 4 minor 68 references

Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPs

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that learning a policy in the abstract latent option space of a continuous hierarchical MDP preserves the optimal value function of the original problem.

desk verdict A novel combination with a broken load-bearing proof: Theorem 11's induction misuses the state-indexed abstraction map, and the paper's own Table 1 contradicts its headline GSM-HARD claim. read the letter →

arxiv 2507.16473 v2 pith:B6SUJ6GU submitted 2025-07-22 cs.AI

Chang Li , Yaren Zhang , Haoran Lv , Qiong Cao , Chao Xue , Xiaodong He This is my paper
classification cs.AI
keywords hierarchicalreinforcementlearningoptionsvariationalinferenceMDPhomomorphismstemporalabstractionlatentreasoningchain-of-thoughtmaximumentropy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that an agent can solve the original hierarchical decision problem by learning in a simplified abstract space of latent options, without any loss of optimality, provided the abstraction is a continuous homomorphism between Hidden Temporal Markov Decision Processes. It introduces VMOC, an off-policy variational option-learning algorithm that represents options as embeddings and maximizes an entropy-augmented objective. The proof centers on Theorem 11, which states that $Q^*(e,a,o) = \bar{Q}^*(f(e), g_e(a,o))$ for a continuous HiT-MDP homomorphism $h=(f,g_e)$, and on Theorem 16, which states that the abstract evidence lower bound is a lower bound of the original evidence lower bound under the lifted policy. The paper also argues that explicit chain-of-thought demonstrations can be distilled into this latent option space during a cold-start supervised fine-tuning phase, giving LLM-style reasoning a structured initialization.

What carries the argument

The object that carries the argument is the continuous HiT-MDP homomorphism $h=(f,g_e)$: $f$ is a vector-bundle map that abstracts the joint state-option space, and $g_e$ is a surjective Borel map, chosen per state $e$, that abstracts the augmented action-option space. Its two defining properties, invariance of reward and equivariance of transitions, are what allow the proof to push optimal $Q$-functions from the original MDP to the abstract MDP. The second mechanism is the variational ELBO with option embeddings: options are rows of a learnable embedding matrix, and optimizing the ELBO produces maximum-entropy exploration because the entropy terms $H[\pi^A]$ and $H[\pi^O]$ appear naturally in the objective. Together these two mechanisms let VMOC train in a simplified space while retaining a formal link to the original problem.

What would settle it

Build a two-state HiT-MDP with overlapping action-option sets and choose surjective abstraction maps $g_{e_1}$ and $g_{e_2}$ that disagree on a shared action. If $Q^*(e_1,a,o) \neq \bar{Q}^*(f(e_1), g_{e_1}(a,o))$ while Definition 8 is satisfied exactly as written, the claimed equality is false.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that the value-equivalence and policy-lifting results known for continuous MDP homomorphisms extend to the option-based HiT-MDP setting, and that this extension justifies variational learning in the abstract space. The authors define a continuous HiT-MDP homomorphism as a bundle map $f$ on the state-option space together with a state-dependent surjective Borel map $g_e$ on action-option pairs, satisfying reward invariance and transition equivariance. They then prove by induction that the optimal action-value function is unchanged by the abstraction, so any policy that is optimal in the abstract latent space corresponds to an optimal policy in the original problem. They further prove that the evidence lower bound optimized by VMOC on the abstract MDP is no larger than the evidence lower bound of the original MDP under the lifted policy, which means abstract-space variational training is a valid surrogate for the original objective.

Load-bearing premise

The proof's load-bearing premise is that the abstraction map chosen at the current state can also relabel actions at the next state, even though the definition only supplies a separate map per state with no requirement that the maps agree.

Editorial extensions

If this is right

  • If Theorem 11 holds, a policy learned in the abstract latent option space can be lifted to the original HiT-MDP with the same optimal action values, so abstract-space training sacrifices no optimality.
  • If Theorem 16 holds, running VMOC on the abstract MDP maximizes a lower bound on the original ELBO, making the abstract variational objective a principled surrogate for the original control problem.
  • Because options are embeddings rather than triples of initiation set, intra-option policy, and termination function, VMOC avoids the usual computational cost of option-critic architectures and scales to large continuous state-action spaces.
  • The cold-start SFT phase gives the latent option space a meaningful initialization from human reasoning demonstrations, so an LLM can start with a library of reasoning primitives and refine them through reinforcement learning rather than learning from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the proof's reliance on a state-dependent abstraction map suggests a testable consistency condition: if $g_e$ is required to agree across states that share actions, Theorem 11's induction becomes robust, and checking whether natural learned abstractions satisfy this condition would separate the theorem's scope from its current definition.
  • Editorial inference: the ELBO inequality $L(q^{\uparrow}) \ge \bar{L}(\bar{q})$ implies a practical monitoring statistic: during VMOC training, tracking the difference between the original and abstract ELBOs could reveal when the learned abstraction drifts outside the homomorphism class.
  • Editorial inference: the cold-start procedure should be compared against a continuous-latent VAE with the same compute; that comparison would isolate whether discrete option structure, rather than latent compression alone, drives the gains on logical reasoning tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes VMOC, an off-policy maximum-entropy option-learning algorithm for HiT-MDPs, together with a cold-start procedure that distills explicit chain-of-thought demonstrations into discrete latent option embeddings for LLM reasoning. The authors also develop a theory of continuous HiT-MDP homomorphisms using vector bundles and claim that learning in the resulting abstract option space preserves optimal value functions (Theorem 11), that lifted policies exist and preserve value (Definition 13, Proposition 14, Theorem 15), and that maximizing the abstract ELBO is a valid surrogate for the original variational objective (Theorem 16). The paper reports experiments on MuJoCo locomotion and on mathematical/logical reasoning benchmarks.

Significance. If the central theoretical claims were correct, the paper would make a valuable contribution: it would give a principled account of when option-based abstractions preserve optimality in continuous state-action spaces and would connect that theory to an off-policy algorithm and to latent reasoning in LLMs. The paper also has practical strengths: it proposes a concrete off-policy algorithm, provides implementation details, ships code in the supplemental material, and evaluates against several baselines in both control and language domains. However, the main theoretical guarantee—optimal value equivalence under continuous HiT-MDP homomorphisms—is not established as written, and the convergence claim for neural-network VMOC rests on an unsupported assertion about stochastic gradient descent. The manuscript therefore does not currently deliver the rigorous foundation it advertises.

major comments (4)
  1. [§5.1, Theorem 11, proof step (19)] The induction step applies the induction hypothesis to the next-state action pair (a_{t+1}, o_{t+1}) using g_{e_t}, the abstraction map attached to the current state e_t=(s_t,o_{t-1}), rather than g_{e_{t+1}}. Definition 8 only supplies, for each e, a surjective Borel map g_e; it imposes no relation between g_{e_t} and g_{e_{t+1}}. Therefore the equality Q_m(e,a,o)=Qbar_m(f(e),g_e(a,o)) is not extended to the next state, and the Bellman recursion in equations (16)-(22) is unjustified. This is the central load-bearing step of the paper, and it invalidates the claim that Q^*(e,a,o)=Qbar^*(f(e),g_e(a,o)). A corrected proof would need an additional consistency condition on the family {g_e}, such as an equivariance relation along transitions; that condition is absent from Definition 8.
  2. [§5.1, Theorem 11, proof step (20)] Even if the map g_{e_t} could be applied at e_{t+1}, the transition from a supremum over g_{e_t}(a_{t+1},o_{t+1}) to a supremum over (\bar a_{t+1}, f_bundle(o_{t+1})) is not justified. Definition 8 requires g_e to be surjective as a map A×O → Abar×Obar, but it does not require its option component to equal f_bundle(o), nor does it require f_bundle to be surjective onto the abstract option space. The change of variables in equation (21) also conflates the abstract state f(e_{t+1}) with e_{t+1} in the notation of Qbar_m. These steps are essential to closing the induction, so the proof of Theorem 11 does not go through.
  3. [§A.4, Theorem 3] The proof of Theorem 3 asserts that 'using stochastic gradient descent (SGD) to optimize the parameters guarantees that the ELBO increases' and then concludes convergence of the sequence {q_k}. This is false in general for neural-network policies: the ELBO is nonconvex, stochastic gradients do not guarantee monotone increase, and convergence to a global or even local optimum is not assured without substantial additional assumptions. Since the theorem is used to justify the convergence of VMOC in Section 3.2, the paper's convergence claim for the practical algorithm is unsupported as stated.
  4. [§6.2, Theorem 16 and following paragraph] Theorem 16 establishes the inequality L(q^↑) ≥ Lbar(q). The subsequent text concludes that 'improving the policy in the abstract space guarantees an improvement in the variational objective of the original problem.' That conclusion does not follow: the gap between L(q^↑) and Lbar(q) is the expected conditional entropy term in equation (37), and this gap can change when q changes. An increase in Lbar(q) therefore does not imply an increase in L(q^↑). Additionally, the proof assumes that the regularizer f is invariant under the homomorphism, but Definition 8 only states reward invariance; the manuscript does not show that the mutual-information regularizer used in Section 3 has this property.
minor comments (4)
  1. [§7.2.4, Table 1] The text states that VMOC-SFT achieves the best score on GSM-HARD with 15.6, but Table 1 lists CoT-SFT at 15.8, which is higher. This empirical claim should be corrected.
  2. [Definition 8] The transition equivariance condition is written as τ_{g_e(a,o)}(B|f(e)) = τ_{(a,o)}(f^{-1}(B)|s), but the argument on the right should presumably be e (or the corresponding state component) rather than s. Please clarify the notation.
  3. [§5.1, equations (16)-(22)] The same symbol Q_m is used for both the original and the abstract value sequence, which makes it difficult to track whether an equality is between corresponding quantities. Using Qbar_m for the abstract MDP would improve readability and prevent apparent circularity.
  4. [§A.4] The proof of Theorem 3 refers to 'assuming the continuity and differentiability of q with respect to its parameters' but does not state which function class is considered; this should be made precise before convergence claims are made.

Circularity Check

1 steps flagged · score 2.0 of 10

No equation-level circularity: the homomorphism and ELBO theorems are genuine derivations, not fits or redefinitions; score reflects minor self-cited ansatz and unproven proof steps.

  1. ansatz smuggled in via citation [Section 3.1, after Eq. 5 (PGM Formulations of The Option Framework)]
    "In this work, we choose f to be the mutual-information f=I[o_t|s_t,a_t,o_{t-1}] as a fact that when the uniform prior assumption of P(o) is relaxed the optimization introduces a mutual-information as a regularizer [44]."

    The paper adopts the mutual-information regularizer as a 'fact' based on a self-citation to the authors' earlier HiT-MDP paper [44], rather than deriving it here. This choice enters the ELBO (Eq. 7) and the option critic objective, so VMOC's objective is partly inherited from prior work by ansatz. The step is not load-bearing for Theorem 11's optimality-preservation claim, which holds for arbitrary non-positive f, so it is a minor self-citation issue rather than a reduction of the central claim.

full rationale

The central derivation is not circular. Theorem 11 is an attempted mathematical proof of optimal value equivalence under HiT-MDP homomorphisms; its inputs are the homomorphism's defining properties (reward invariance and transition equivariance) and the Bellman recursion, not the conclusion. Theorem 16's ELBO lower bound is a standard entropy decomposition, and its reward/regularizer term matching is definitional once the abstract regularizer is chosen to be consistent with the original. No fitted parameter is renamed as a prediction, and no claim reduces by construction to its own hypothesis. The proof of Theorem 11 does contain an unsupported step: equation (19) abstracts the next-state action pair with g_{e_t}, the map attached to the current state, instead of g_{e_{t+1}}, and step (20) restricts the supremum to f_bundle(o_{t+1}) based only on surjectivity of g_{e_t}. Similarly, Theorem 16 assumes the regularizer is homomorphism-invariant even though Definition 8 only requires reward invariance. These are correctness gaps or missing premises, not circularity; they do not make the derivation equivalent to its inputs. The empirical text also says VMOC-SFT 'achieves the best score on GSM-HARD with 15.6', while Table 1 shows CoT-SFT at 15.8, a reporting error rather than a circular step. The only notable self-citation is the mutual-information regularizer adopted from the authors' prior work [44]; it is used in the VMOC objective but is not load-bearing for the optimality-preservation theorem. Overall, the paper's central homomorphism claim has independent content, so the circularity score is low.

Assumptions & free parameters 4 free parameters · 4 assumptions · 3 invented entities

The central theoretical results rest on the HiT-MDP equivalence assumed from the authors' prior work, a vector-bundle structure asserted on the state-option space, and the control-as-inference optimality likelihoods; none are independently established in this paper. At the algorithmic level, entropy temperatures, option counts, KL weight, and embedding sizes are hand-chosen, and Theorem 3's convergence relies on an unproven monotonicity of SGD on the ELBO.

free parameters (4)
  • Entropy temperatures alpha_A and alpha_O = 0.05 fixed in MuJoCo; also updated by gradient in SAC-style objectives
    Controls the balance between reward and entropy in the maximum-entropy objective, Eq. (9) and Section 7.1.
  • Number of options K = 4 for MuJoCo, 6 for LLM cold-start
    Hand-chosen following option literature conventions for MuJoCo and arbitrary for LLM; central to the option embedding matrix W.
  • KL weight beta = 0.1
    Weight on KL regularization in the cold-start ELBO, Algorithm 1; chosen without sensitivity analysis.
  • Option embedding dimension = 40 for MuJoCo, latent vocabulary size 6 for LLM
    Embedding matrix W in R^{4x40}; the LLM latent code length L is unspecified, making the effective latent size unclear.
assumptions (4)
  • domain assumption The option-induced HiT-MDP is homomorphically equivalent to the conventional SMDP option framework.
    Invoked throughout Section 3 to justify learning options as temporal embeddings; inherited from the authors' prior work [44,45] without independent formal or empirical verification in this paper.
  • ad hoc to paper State-option space E_O is a vector bundle over the state manifold S with option spaces as fibers of possibly different dimensions.
    Introduced in Definition 4 and Remark 5 to carry over continuous MDP homomorphism theory [48]; it is a mathematical modeling choice with no falsifiable handle, and it conflicts with the discrete option embeddings used in the experiments.
  • domain assumption Optimality likelihoods are P(E_t^A | s,a) = exp(r(s,a)) and P(E_t^O | s,a,o,o') = exp(f(·)) with f chosen as mutual information.
    Standard control-as-inference assumption, Eq. (4)-(5); the specific choice f = I[o_t | s_t, a_t, o_{t-1}] is imported from [44].
  • ad hoc to paper Stochastic gradient descent on the ELBO monotonically increases the evidence lower bound and converges for neural-network policies.
    Assumed in the proof of Theorem 3 (Appendix A.4); this is not true in general for nonconvex objectives and would require step-size and architecture conditions that the paper does not state.
invented entities (3)
  • Continuous vector bundle E_O over the state manifold with option spaces as fibers
    purpose: Provides the geometric setting for continuous HiT-MDP homomorphisms and the value-equivalence proofs
    No independent falsifiable prediction; it is an auxiliary mathematical structure that the discrete option experiments do not use.
  • Latent option token vocabulary as implicit chain-of-thought
    purpose: Cold-start SFT maps prompts to a small discrete latent space and generates answers without explicit reasoning text
    No separate test of the latent space semantics; benchmark accuracy is mixed and does not confirm that the options encode human-like reasoning steps.
  • Discrete optimality variables E^A_t and E^O_t in the option PGM
    purpose: Casts optimal option trajectories as probabilistic inference to derive maximum-entropy objectives
    Latent variables standard in control-as-inference; not directly observable or independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPs." pith.science (2026). https://pith.science/paper/B6SUJ6GU

@misc{pith2026250716473,
  author       = {Pith},
  title        = {Pith review of: Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B6SUJ6GU}},
  note         = {Machine review of arXiv:2507.16473}
}
read the original abstract

Large Language Models (LLMs) have shown remarkable reasoning ability through explicit Chain-of-Thought (CoT) prompting, but generating these step-by-step textual explanations is computationally expensive and slow. To overcome this, we aim to develop a framework for efficient, implicit reasoning, where the model "thinks" in a latent space without generating explicit text for every step. We propose that these latent thoughts can be modeled as temporally-extended abstract actions, or options, within a hierarchical reinforcement learning framework. To effectively learn a diverse library of options as latent embeddings, we first introduce the Variational Markovian Option Critic (VMOC), an off-policy algorithm that uses variational inference within the HiT-MDP framework. To provide a rigorous foundation for using these options as an abstract reasoning space, we extend the theory of continuous MDP homomorphisms. This proves that learning a policy in the simplified, abstract latent space, for which VMOC is suited, preserves the optimality of the solution to the original, complex problem. Finally, we propose a cold-start procedure that leverages supervised fine-tuning (SFT) data to distill human reasoning demonstrations into this latent option space, providing a rich initialization for the model's reasoning capabilities. Extensive experiments demonstrate that our approach achieves strong performance on complex logical reasoning benchmarks and challenging locomotion tasks, validating our framework as a principled method for learning abstract skills for both language and control.

Figures

Figures reproduced from arXiv: 2507.16473 by the authors.

Figure 1
Figure 1. PGMs of the option framework. by introducing the concept of "Optimality" [65] into the conventional SMDP-based option framework (Equation equation 1). This allows us to define the probability of an option trajectory being optimal as a probabilistic graphical model (PGM), as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. The training process for the cold-start phase. A complete data sample (prompt, CoT, answer) is fed into a shared encoder to produce a posterior distribution. A latent option is sampled and used to decode both the CoT and the answer, providing reconstruction signals. The KL divergence between the posterior and the prior (generated from the prompt only) acts as a regularizer. All components are trained jointly with a … view at source ↗
Figure 3
Figure 3. The inference process. With no access to the ground truth reasoning or answer, the model first generates a distribution over latent options from the prior network given only the input prompt. A latent option is sampled and then used to autoregressively generate the final answer. Algorithm 1 Cold-Start Training for Latent Reasoning 1: Input: SFT dataset D, learning rate η, KL weight β 2: Initialize: Option embeddings… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The only exception is the relatively simple InvertedDoublePendulum environment, which we suspect is due to hyper-parameter tuning issues and will be addressed in future work [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 43 canonical work pages

  1. [1]

    Opal: Offline primitive discovery for accelerating offline reinforcement learning.arXiv preprint arXiv:2010.13611, 2020

    Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. Opal: Offline primitive discovery for accelerating offline reinforcement learning.arXiv preprint arXiv:2010.13611, 2020

    arXiv 2010

  2. [2]

    Araujo, E. G. and Grupen, R. A. Learning control composition in a complex environment. In Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior, pp. 333–342, 1996

    work page 1996

  3. [3]

    PhD thesis, McGill University Libraries, 2018

    Bacon, P.-L.Temporal Representation Learning. PhD thesis, McGill University Libraries, 2018

    work page 2018

  4. [4]

    The option-critic architecture

    Bacon, P.-L., Harb, J., and Precup, D. The option-critic architecture. InThirty-First AAAI Conference on Artificial Intelligence, 2017

    work page 2017

  5. [5]

    and Tsitsiklis, J

    Bertsekas, D. and Tsitsiklis, J. N.Neuro-dynamic programming. Athena Scientific, 1996

    work page 1996

  6. [6]

    Brockett, R. W. Hybrid models for motion control systems. InEssays on Control: Perspectives in the Theory and its Applications, pp. 29–53. Springer, 1993

    work page 1993

  7. [7]

    Openai gym.arXiv preprint arXiv:1606.01540, 2016

    Brockman, G., Cheung, V ., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym.arXiv preprint arXiv:1606.01540, 2016

    arXiv 2016

  8. [8]

    Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings

    Co-Reyes, J., Liu, Y ., Gupta, A., Eysenbach, B., Abbeel, P., and Levine, S. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. In International conference on machine learning, pp. 1009–1018. PMLR, 2018

    work page 2018

Show all 68 references
  1. [9]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  2. [10]

    Behavior analysis and training-a methodology for behavior engineering.IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 26(3):365–380, 1996

    Colombetti, M., Dorigo, M., and Borghi, G. Behavior analysis and training-a methodology for behavior engineering.IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 26(3):365–380, 1996

    1996

  3. [11]

    Probabilistic inference for determining options in reinforcement learning.Machine Learning, 104(2-3):337–357, 2016

    Daniel, C., Van Hoof, H., Peters, J., and Neumann, G. Probabilistic inference for determining options in reinforcement learning.Machine Learning, 104(2-3):337–357, 2016

    2016

  4. [12]

    and Hinton, G

    Dayan, P. and Hinton, G. E. Feudal reinforcement learning.Advances in Neural Information Processing Systems, pp. 271–278, 1993

    1993

  5. [13]

    Implicit chain of thought reasoning via knowledge distillation.arXiv preprint arXiv:2311.01460, 2023

    Deng, Y ., Prasad, K., Fernandez, R., Smolensky, P., Chaudhary, V ., and Shieber, S. Implicit chain of thought reasoning via knowledge distillation.arXiv preprint arXiv:2311.01460, 2023

    2023 arXiv

  6. [14]

    From explicit cot to implicit cot: Learning to internalize cot step by step.arXiv preprint arXiv:2405.14838, 2024

    Deng, Y ., Choi, Y ., and Shieber, S. From explicit cot to implicit cot: Learning to internalize cot step by step.arXiv preprint arXiv:2405.14838, 2024

    2024 arXiv

  7. [15]

    Dietterich, T. G. Hierarchical reinforcement learning with the maxq value function decomposi- tion.Journal of Artificial Intelligence Research, 13:227–303, 2000

    2000

  8. [16]

    Diversity is all you need: Learning skills without a reward function.arXiv preprint arXiv:1802.06070, 2018

    Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without a reward function.arXiv preprint arXiv:1802.06070, 2018

    2018 arXiv

  9. [17]

    Addressing function approximation error in actor-critic methods.arXiv preprint arXiv:1802.09477, 2018

    Fujimoto, S., Van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods.arXiv preprint arXiv:1802.09477, 2018

    2018 arXiv

  10. [18]

    Pal: Program-aided language models

    Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y ., Callan, J., and Neubig, G. Pal: Program-aided language models. InInternational Conference on Machine Learning, pp. 10764– 10799. PMLR, 2023

    2023

  11. [19]

    R., Kailkhura, B., Bhatele, A., and Goldstein, T

    Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., Kailkhura, B., Bhatele, A., and Goldstein, T. Scaling up test-time compute with latent reasoning: A recurrent depth approach.arXiv preprint arXiv:2502.05171, 2025

    2025 arXiv

  12. [20]

    Infobot: Transfer and exploration via the information bottleneck.arXiv preprint arXiv:1901.10902, 2019

    Goyal, A., Islam, R., Strouse, D., Ahmed, Z., Botvinick, M., Larochelle, H., Bengio, Y ., and Levine, S. Infobot: Transfer and exploration via the information bottleneck.arXiv preprint arXiv:1901.10902, 2019. 22

    1901 arXiv

  13. [21]

    S., Menon, A

    Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., and Nagarajan, V . Think before you speak: Training language models with pause tokens.arXiv preprint arXiv:2310.02226, 2023

    2023 arXiv

  14. [22]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  15. [23]

    S., and Brunskill, E

    Guo, Z., Thomas, P. S., and Brunskill, E. Using options and covariance testing for long horizon off-policy policy evaluation. InAdvances in Neural Information Processing Systems, pp. 2492–2501, 2017

    2017

  16. [24]

    Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning.arXiv preprint arXiv:1910.11956, 2019

    Gupta, A., Kumar, V ., Lynch, C., Levine, S., and Hausman, K. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning.arXiv preprint arXiv:1910.11956, 2019

    1910 arXiv

  17. [25]

    Reinforcement learning with deep energy- based policies

    Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy- based policies. InInternational Conference on Machine Learning, pp. 1352–1361. PMLR, 2017

    2017

  18. [26]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.arXiv preprint arXiv:1801.01290, 2018

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.arXiv preprint arXiv:1801.01290, 2018

    2018 arXiv

  19. [27]

    Training large language models to reason in a continuous latent space, 2024.URL https://arxiv

    Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., and Tian, Y . Training large language models to reason in a continuous latent space, 2024.URL https://arxiv. org/abs/2412.06769, 2022

    2024 arXiv

  20. [28]

    When waiting is not an option: Learning options with a deliberation cost

    Harb, J., Bacon, P.-L., Klissarov, M., and Precup, D. When waiting is not an option: Learning options with a deliberation cost. InThirty-Second AAAI Conference on Artificial Intelligence, 2018

    2018

  21. [29]

    The termination critic.arXiv preprint arXiv:1902.09996, 2019

    Harutyunyan, A., Dabney, W., Borsa, D., Heess, N., Munos, R., and Precup, D. The termination critic.arXiv preprint arXiv:1902.09996, 2019

    1902 arXiv

  22. [30]

    T., Wang, Z., Heess, N., and Riedmiller, M

    Hausman, K., Springenberg, J. T., Wang, Z., Heess, N., and Riedmiller, M. Learning an embedding space for transferable robot skills. InInternational Conference on Learning Repre- sentations, 2018

    2018

  23. [31]

    and Precup, D

    Kamat, A. and Precup, D. Diversity-enriched option-critic.arXiv, 2020

    2020

  24. [32]

    and Precup, D

    Khetarpal, K. and Precup, D. Learning options with interest functions. InProceedings of the 32nd AAAI Conference on Artificial Intelligence, pp. 1–2, 2019

    2019

  25. [33]

    Options of interest: Temporal abstraction with interest functions

    Khetarpal, K., Klissarov, M., Chevalier-Boisvert, M., Bacon, P.-L., and Precup, D. Options of interest: Temporal abstraction with interest functions. InProceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4,444–4,451, 2020

    2020

  26. [34]

    and Precup, D

    Klissarov, M. and Precup, D. Flexible option learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y ., Liang, P., and Vaughan, J. W. (eds.),Advances in Neural Information Processing Systems, volume 34, pp. 4632–4646. Curran Associates, 2021

    2021

  27. [35]

    Learnings options end-to-end for continu- ous action tasks.arXiv preprint arXiv:1712.00004, 2017

    Klissarov, M., Bacon, P.-L., Harb, J., and Precup, D. Learnings options end-to-end for continu- ous action tasks.arXiv preprint arXiv:1712.00004, 2017

    2017 arXiv

  28. [36]

    and Friedman, N.Probabilistic graphical models: principles and techniques

    Koller, D. and Friedman, N.Probabilistic graphical models: principles and techniques. MIT press, 2009

    2009

  29. [37]

    S., et al

    Kolobov, A., Weld, D. S., et al. Discovering hidden structure in factored mdps.Artificial Intelligence, 189:19–47, 2012

    2012

  30. [38]

    and Barto, A

    Konidaris, G. and Barto, A. G. Skill discovery in continuous reinforcement learning domains using skill chaining. InAdvances in neural information processing systems, pp. 1015–1023, 2009. 23

    2009

  31. [39]

    Reinforcement learning and control as probabilistic inference: Tutorial and review

    Levine, S. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018

    2018 arXiv

  32. [40]

    Levy, K. Y . and Shimkin, N. Unified inter and intra options learning using policy gradient methods. InEuropean Workshop on Reinforcement Learning, pp. 153–164. Springer, 2011

    2011

  33. [41]

    Soac: The soft option actor-critic architecture.arXiv preprint arXiv:2006.14363, 2020

    Li, C., Ma, X., Zhang, C., Yang, J., Xia, L., and Zhao, Q. Soac: The soft option actor-critic architecture.arXiv preprint arXiv:2006.14363, 2020

    2006 arXiv

  34. [43]

    The skill-action architecture: Learning abstract action embeddings for reinforcement learning.OpenReview:id=PU35uLgRZkk, 2020

    Li, C., Song, D., and Tao, D. The skill-action architecture: Learning abstract action embeddings for reinforcement learning.OpenReview:id=PU35uLgRZkk, 2020

    2020

  35. [44]

    Hit-mdp: learning the smdp option framework on mdps with hidden temporal embeddings

    Li, C., Song, D., and Tao, D. Hit-mdp: learning the smdp option framework on mdps with hidden temporal embeddings. InThe Eleventh International Conference on Learning Representations, 2022

    2022

  36. [45]

    Hit-mdp: learning the smdp option framework on mdps with hidden temporal embeddings

    Li, C., Song, D., and Tao, D. Hit-mdp: learning the smdp option framework on mdps with hidden temporal embeddings. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  37. [46]

    A., Veness, J., Bellemare, M

    Mnih, V ., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning.Nature, 518(7540):529–533, 2015

    2015

  38. [47]

    Hierarchical reinforcement learning via advantage- weighted information maximization.arXiv preprint arXiv:1901.01365, 2019

    Osa, T., Tangkaratt, V ., and Sugiyama, M. Hierarchical reinforcement learning via advantage- weighted information maximization.arXiv preprint arXiv:1901.01365, 2019

    1901 arXiv

  39. [48]

    Policy gradient methods in the presence of symmetries and state abstractions.Journal of Machine Learning Research, 25(71):1–57, 2024

    Panangaden, P., Rezaei-Shoshtari, S., Zhao, R., Meger, D., and Precup, D. Policy gradient methods in the presence of symmetries and state abstractions.Journal of Machine Learning Research, 25(71):1–57, 2024

    2024

  40. [49]

    Patel, A., Bhattamishra, S., and Goyal, N. Are NLP models really able to solve simple math word problems? In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y . (eds.),Proceedings of the 2021 C...

    2021 doi

  41. [50]

    Long-horizon visual planning with goal-conditioned hierarchical predictors.NeurIPS, 2020

    Pertsch, K., Rybkin, O., Ebert, F., Finn, C., Jayaraman, D., and Levine, S. Long-horizon visual planning with goal-conditioned hierarchical predictors.NeurIPS, 2020

    2020

  42. [51]

    University of Massachusetts Amherst, 2000

    Precup, D.Temporal abstraction in reinforcement learning. University of Massachusetts Amherst, 2000

    2000

  43. [52]

    and Roth, D

    Roy, S. and Roth, D. Solving general arithmetic word problems. In Màrquez, L., Callison- Burch, C., and Su, J. (eds.),Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1743–1752, Lisbon, Portugal, September 2015. Association for Comput...

    2015 doi

  44. [53]

    J., and Hashimoto, T

    Ruan, Y ., Band, N., Maddison, C. J., and Hashimoto, T. Reasoning to learn from latent thoughts. arXiv preprint arXiv:2503.18866, 2025

    2025

  45. [54]

    Equivalence between policy gradients and soft q-learning

    Schulman, J., Chen, X., and Abbeel, P. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440, 2017

    2017 arXiv

  46. [55]

    Proximal policy optimiza- tion algorithms.arXiv preprint arXiv:1707.06347, 2017

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimiza- tion algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  47. [56]

    Dynamics-aware unsupervised discovery of skills.arXiv preprint arXiv:1907.01657, 2019

    Sharma, A., Gu, S., Levine, S., Kumar, V ., and Hausman, K. Dynamics-aware unsupervised discovery of skills.arXiv preprint arXiv:1907.01657, 2019. 24

    1907 arXiv

  48. [58]

    Codi: Compressing chain-of-thought into continuous space via self-distillation.arXiv preprint arXiv:2502.21074, 2025

    Shen, Z., Yan, H., Zhang, L., Hu, Z., Du, Y ., and He, Y . Codi: Compressing chain-of-thought into continuous space via self-distillation.arXiv preprint arXiv:2502.21074, 2025

    2025 arXiv

  49. [59]

    Taco: Learning task decomposition via temporal alignment for control

    Shiarlis, K., Wulfmeier, M., Salter, S., Whiteson, S., and Posner, I. Taco: Learning task decomposition via temporal alignment for control. InInternational Conference on Machine Learning, pp. 4654–4663. PMLR, 2018

    2018

  50. [60]

    J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V ., Lanctot, M., et al

    Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V ., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search.Nature, 529(7587):484–489, 2016

    2016

  51. [61]

    An inference-based policy gradient method for learning options

    Smith, M., Hoof, H., and Pineau, J. An inference-based policy gradient method for learning options. InInternational Conference on Machine Learning, pp. 4,703–4,712, 2018

    2018

  52. [62]

    Sutton, R. S. and Barto, A. G.Reinforcement learning: An introduction. MIT press, 2018

    2018

  53. [63]

    S., Precup, D., and Singh, S

    Sutton, R. S., Precup, D., and Singh, S. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.Artificial Intelligence, 112(1-2):181–211, 1999

    1999

  54. [64]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    Talmor, A., Herzig, J., Lourie, N., and Berant, J. Commonsenseqa: A question answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    2018 arXiv

  55. [65]

    Linearly-solvable markov decision problems.Advances in neural information processing systems, 19, 2006

    Todorov, E. Linearly-solvable markov decision problems.Advances in neural information processing systems, 19, 2006

    2006

  56. [66]

    V ., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V ., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  57. [67]

    Data-efficient hindsight off-policy option learning

    Wulfmeier, M., Rao, D., Hafner, R., Lampe, T., Abdolmaleki, A., Hertweck, T., Neunert, M., Tirumala, D., Siegel, N., Heess, N., et al. Data-efficient hindsight off-policy option learning. arXiv preprint arXiv:2007.15588, 2020

    2007 arXiv

  58. [68]

    Zhang, D., Courville, A., Bengio, Y ., Zheng, Q., Zhang, A., and Chen, R. T. Latent state marginalization as a low-cost approach for improving exploration.arXiv preprint arXiv:2210.00999, 2022

    2022 arXiv

  59. [69]

    and Whiteson, S

    Zhang, S. and Whiteson, S. DAC: The double actor-critic architecture for learning options. In Advances in Neural Information Processing Systems, pp. 2,012–2,022, 2019

    2019

  60. [70]

    D., Bagnell, J

    Ziebart, B. D., Bagnell, J. A., and Dey, A. K. Modeling interaction via the principle of maximum causal entropy. InICML, 2010. A Proofs A.1 Theorem 1 Theorem 1(Convergence Theorem for Structured Variational Policy Iteration).Let τ be the latent variable and E be the observed v...

    2010

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.