Pith. sign in

REVIEW 4 major objections 5 minor 62 references

Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidance

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a hierarchical prompting scheme, with a global token summarizing task dynamics and adaptive tokens retrieved from demonstrations, lets a decision transformer generalize to new tasks from a few demonstrations and…

desk verdict HPDT is a sensible new combination, but the paper's own ablation shows the headline gains are mostly Time2Vec, not hierarchical prompting. read the letter →

arxiv 2412.00979 v2 pith:WRLPPCMF submitted 2024-12-01 cs.LG

Zhe Wang , Haozhu Wang , Yanjun Qi This is my paper · ORCID
classification cs.LG
keywords few-shotpolicygeneralizationofflinemeta-reinforcementlearningdecisiontransformerhierarchicalpromptingretrievalaugmentationsoftprompttokensin-context
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a decision transformer can generalize to a new reinforcement-learning task from a handful of demonstrations if it receives two levels of prompt: a global token that summarizes the task's transition dynamics and reward pattern, and adaptive tokens that are retrieved from demonstration segments at each timestep. Current prompting methods for decision transformers use randomly sampled, static prompt segments, so they give the same guidance everywhere and often fail to identify the task. HPDT replaces those static prompts with learned soft tokens and a KNN retriever, and reports that this scheme achieves the best comparison results on six of seven MuJoCo and MetaWorld benchmarks, with the global token mattering most when task identities are discrete and the adaptive tokens mattering most when task identities vary continuously. A reader should care because it suggests that in-context prompting, without gradient updates, can be a competitive route to few-shot policy generalization.

What carries the argument

The central object is the two-tier soft prompt. The global token $g^z_{\mathcal{M}_i}$ is a single vector formed by the mean over the demonstration trajectory of $\sigma(h_{\theta_g}([\hat{r}^*_{i,t}, s^*_{i,t}, a^*_{i,t}, s^*_{i,t+1}, \hat{r}^*_{i,t+1}]))$, a GELU-activated linear projection of each transition tuple; it is meant to encode the task's transition dynamics and reward function. The adaptive tokens are produced by, for each rollout timestep $t$, taking the $k$ demonstration tuples whose return-to-go/state pairs are nearest in Euclidean distance to the current pair, and averaging their projected return-to-go, state, and action tokens. These adaptive tokens are fused with the rollout tokens by addition so that no new modality is introduced into the sequence, and a Time2Vec embedding provides relative time information with far fewer parameters than a lookup table. The causal transformer is trained end-to-end on the augmented sequences with action MSE, so the prompt layers and time embedding are learned together with the policy.

What would settle it

Train on two task families whose per-timestep transition tuples have identical means but differ in, for example, reward variance or the shape of the next-state distribution, then test whether HPDT distinguishes them during few-shot rollout; if it cannot, the mean-pooled global token is the reason.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that task identity and timestep-specific context can be separated into two soft-prompt mechanisms and that this separation helps a decision transformer adapt in offline meta-RL. The global token is computed by averaging, over one sampled demonstration trajectory, the projected transition tuples (current remaining reward, state, action, next state, next remaining reward); this average is intended to be a permutation-invariant summary of the transition dynamics and reward function. The adaptive tokens are computed for each rollout timestep by retrieving the closest demonstration tuples to the current remaining-reward/state pair and averaging their projections. When these tokens are prepended or added into the transformer input, the resulting model, HPDT, is reported to outperform MACAW, PDT, PTDT, and PDT-FT on six of the seven evaluation environments, and the abstract states the broader claim that it consistently outperforms all baselines across all seven.

Load-bearing premise

The method relies on the average of projected transition tuples from one demonstration trajectory capturing enough of a task's identity; if two tasks have the same average but differ in other aspects of their dynamics or rewards, the global token cannot tell them apart.

Editorial extensions

If this is right

  • HPDT's global token compresses task identity into one vector, so the prompt length no longer grows with the number of demonstration timesteps needed to convey transition dynamics and rewards.
  • Adaptive tokens provide additional per-timestep guidance that matters most when task identities are similar, such as continuously varying goal velocities or goal positions.
  • The method's in-context prompting outperforms full model fine-tuning (PDT-FT) in the reported benchmarks, suggesting gradient-free adaptation can be sufficient for few-shot offline meta-RL.
  • Performance stays comparable when the demonstration length shrinks or the KNN neighborhood size changes, indicating the method does not require careful tuning of those hyperparameters.
  • The Time2Vec time embedding accelerates early training convergence relative to lookup-table embeddings because it has fewer parameters and is adjacency-aware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the global-token mean is sufficient for task identity, the same two-tier prompting could be dropped into other sequence-conditioned policies, such as trajectory or diffusion planners, by projecting their transition tuples.
  • The Euclidean KNN retriever in raw projected space is likely not the only viable choice; a learned or differentiable retriever could improve adaptive tokens on high-dimensional tasks, and a testable comparison would show whether retrieval quality is the bottleneck.
  • The single global token from one sampled demonstration may be brittle when demonstrations are noisy or when reward distributions differ in higher moments; using multiple demonstration trajectories or a learned set aggregator would be a natural stress test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes the Hierarchical Prompt Decision Transformer (HPDT), a method for offline meta-reinforcement learning that augments a Decision Transformer with two levels of soft prompts: a global token obtained by mean-pooling projected transition tuples from a demonstration trajectory (Eq. 4), and adaptive tokens obtained by KNN retrieval of similar rtg-state-action tuples that are added to the rollout tokens (Eqs. 6-8). The paper also replaces the lookup-table time embedding used by PDT with Time2Vec (Eq. 9). Experiments on five MuJoCo and two MetaWorld benchmarks compare HPDT with MACAW, PDT, PDT-FT, and PTDT, report average returns over three random seeds, and include ablations (HPDT w/o G, HPDT w/o A, HPDT w/o T) plus a hyperparameter robustness study. The central claim is that hierarchical global-plus-adaptive prompting consistently outperforms all baselines and is an effective few-shot generalization strategy.

Significance. If the causal contribution of hierarchical prompting were cleanly established, the method would be a useful addition to the OMRL literature: the global-token design is conceptually simple, the adaptive retrieval mechanism is plausible, and the paper includes useful ablations, a robustness table, and an inference-cost analysis. The evaluation uses standard held-out benchmarks and I see no circularity: test-task rollouts are not used to fit parameters. However, the headline claim of 'consistently outperforms all baseline methods' is not supported by the paper's own tables, and the attribution of the empirical gains to hierarchical prompting rather than to the simultaneous switch to Time2Vec is not established by the reported experiments. This is the load-bearing weakness of the paper.

major comments (4)
  1. [§4, Table 4 vs. Table 3] The central causal claim that hierarchical prompting is effective is confounded with the time-encoding change. Full HPDT differs from PDT in two respects: it adds the hierarchical global/adaptive tokens and it replaces PDT's lookup-table time embedding with Time2Vec. The ablation HPDT w/o T keeps the hierarchical tokens while reverting to the lookup-table embedding. On Pick&Place, HPDT w/o T scores 432.5±19.0, which is below PDT's 497.5±34.8 in Table 3; on Cheetah-Vel and Cheetah-Dir the isolated hierarchy gains are only about +6.6 and +8.4. Thus the large reported gains over PDT, especially the +72.0 on Pick&Place, are largely or entirely attributable to Time2Vec rather than to the hierarchical prompting mechanism. To support the abstract's conclusion, the authors should report a PDT+Time2Vec baseline that changes only the time embedding while keeping PDT's static prompt, and should compare it against HPDT and HPDT w/o T.
  2. [Abstract and §4, Table 3] The statement that HPDT 'consistently outperforms all baseline methods' is contradicted by Table 3. On Param-Walker, PTDT achieves 368.9±21.9 while HPDT achieves 367.3±32.5, so HPDT is not the best method there. On Reach, HPDT's 4832.2±5.2 is within one standard deviation of PTDT's 4830.5±2.9 and PDT-FT's 4828.3±6.5. The Section 4 claim of 'optimal results ... on six out of seven meta-environments' is accurate only if Param-Walker is the exception, but this should be stated explicitly in the abstract and conclusion rather than using 'consistently outperforms all baselines'. The claims should be tempered to match the evidence.
  3. [§4, Tables 3-4, Figure 2] The paper relies on only three random seeds and reports no significance tests or confidence intervals, yet several comparisons that feed the headline claims are within noise. For example, on Pick&Place, HPDT w/o G (568.0±5.5) and HPDT (569.5±5.1) are essentially tied, and on Reach the baselines overlap with HPDT. The authors should either provide more seeds, report paired significance tests, or explicitly identify which comparisons are robust. As it stands, the repeated wording 'significant improvements' and 'consistent improvements' is not supported by the reported statistics.
  4. [§3, Eq. (4)] The global token is defined as a mean over projected transition tuples, so the method implicitly assumes that the mean is a sufficient statistic for task identity. If tasks differ in higher-order statistics of the reward or transitions (e.g., variance or skew rather than mean), the mean-pooled global token cannot disambiguate them and the method would degrade toward the HPDT w/o G ablation. Figure 4 shows qualitative clustering but does not measure how much task-discriminative information survives pooling. I would ask the authors to quantify this, for example by reporting nearest-neighbor task classification accuracy on the global tokens, or by ablating the aggregator (mean vs. attention vs. max) on a task set where reward variance differs. This is a correctness-risk concern rather than a demonstrated failure, but it should be addressed or explicitly discussed as a limitation.
minor comments (5)
  1. [§3, Evaluation paragraph] The sentence 'we randomly sample a demonstration trajectory Segment_z(O_M_i)' should refer to the demonstration set D_M_j of the new task, not the rollout set O_M_i; the notation is inconsistent with Section 2.
  2. [References] Reference [13] is cited as PTDT and described as 'applied the zeroth-order optimization-based approach for prompt tuning', but the title 'Prompt-Tuning Decision Transformer with Preference Ranking' suggests the method is about preference ranking; if this is the intended PTDT baseline, the description should be aligned with the cited work, or the correct reference should be given.
  3. [Figure 3 caption] The caption contains a duplicated label '3(d)(d)' in the second paragraph; it should read 'Figure 3(d)' and 'Figure 3(e)'.
  4. [§4, Ablation paragraph] The text alternates between 'HPDT wo A' and 'HPDT w/o A' (similarly for G and T). Please use one consistent notation throughout the paper and figures.
  5. [§4, Table 4] For Pick&Place, the authors emphasize that adaptive tokens are important, but Table 4 shows HPDT w/o A (499.0±33.7) is essentially at the PDT level (497.5±34.8) rather than clearly below the full model; the discussion should acknowledge that the adaptive-token benefit on this environment is also partly entangled with the Time2Vec change.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity is present: the evaluation is a held-out benchmark, and the paper's self-citations are not load-bearing.

full rationale

The paper's central claim is empirical. HPDT's global token (Eq. 4) and adaptive tokens (Eq. 6) are explicit forward operations on sampled demonstration trajectories, and all learnable parameters are fit by MSE on training-task rollout actions (Algorithm 1). Evaluation (Section 4, Table 3) uses unseen test tasks with target-return constants shared with baselines, so the reported returns cannot be forced by construction from the inputs. The only self-citations are [10] for Time2Vec (co-cited with the external original [18]) and [47] for the mean/set aggregator (co-cited with Deep Sets [52]); neither is load-bearing because the formulas are stated in Eqs. (4) and (9) and are standard. The skeptical observation that HPDT's gain over PDT may be confounded with the Time2Vec change (HPDT wo T scores 432.5 vs PDT 497.5 on Pick&Place) is a correctness or attribution concern, not a circularity, so it does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The central claim rests on standard OMRL assumptions and on the design assumption that mean-pooled transition tuples and KNN retrieval carry the task information needed for few-shot generalization. No physical entities are introduced; the 'entities' are learned embeddings with no external falsifiable handle.

free parameters (3)
  • k (number of KNN neighbors) = 3 (default), varied 3/5/7/10
    Number of retrieved rtg-state-action tuples averaged into adaptive tokens in Eq. (6). Chosen by hand; robustness is tested but the default drives main results.
  • m' (demonstration segment length) = 20 (default), varied 10/25
    Number of timesteps in the demonstration segment used for global and adaptive token learning. Chosen by hand; Table 5 varies it.
  • Target return-to-go values per environment = e.g., 1500 (Cheetah-dir), 0 (Cheetah-vel), etc.
    Fixed reward-conditioning targets in Table 2, chosen per environment and shared across methods. They affect the conditioning signal but not the comparison.
assumptions (5)
  • domain assumption OMRL tasks share state and action spaces and differ only in transition dynamics and reward functions.
    Section 2 states this as the OMRL assumption. If tasks differ in other ways, the meta-learning setup breaks.
  • domain assumption A mean of projected transition tuples is a sufficient statistic for task identity.
    Section 3, Eq. (4) uses mean pooling to derive the global token. This is the load-bearing assumption identified in weakest_assumption.
  • domain assumption Euclidean distance between rtg-state pairs ranks relevant demonstration segments correctly.
    Section 3, Learning Adaptive Tokens, uses KNN with Euclidean distance on raw rtg-state pairs. If this similarity is uninformative, the retrieved tokens add noise.
  • domain assumption The demonstration set D_{M_j} is representative of the test task.
    The paper calls the demonstrations 'curated' in the abstract but provides no curation protocol; the method samples one trajectory randomly, so representativeness is assumed.
  • standard math Causal transformer can condition on prepended and summed tokens without modality confusion.
    Section 3, integration of adaptive tokens via summation rather than concatenation assumes the transformer's embeddings can mix modalities; this is an architectural assumption inherited from DT/PDT.
invented entities (2)
  • Global token g^z
    purpose: Learned embedding intended to summarize task-level transition dynamics and reward function; prepended to the rollout sequence.
    It is an internal parameter of HPDT. The paper shows 2D projections (Figure 4) but no external falsifiable prediction tied to the token itself.
  • Adaptive tokens (retrieved template sequence)
    purpose: Per-timestep guidance generated by KNN retrieval from the demonstration and summed into rollout tokens.
    Internal conditioning signals; their effect is only observable through benchmark returns, not through an independent measurement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidance." pith.science (2026). https://pith.science/paper/WRLPPCMF

@misc{pith2026241200979,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WRLPPCMF}},
  note         = {Machine review of arXiv:2412.00979}
}
read the original abstract

Decision transformers recast reinforcement learning as a conditional sequence generation problem, offering a simple but effective alternative to traditional value or policy-based methods. A recent key development in this area is the integration of prompting in decision transformers to facilitate few-shot policy generalization. However, current methods mainly use static prompt segments to guide rollouts, limiting their ability to provide context-specific guidance. Addressing this, we introduce a hierarchical prompting approach enabled by retrieval augmentation. Our method learns two layers of soft tokens as guiding prompts: (1) global tokens encapsulating task-level information about trajectories, and (2) adaptive tokens that deliver focused, timestep-specific instructions. The adaptive tokens are dynamically retrieved from a curated set of demonstration segments, ensuring context-aware guidance. Experiments across seven benchmark tasks in the MuJoCo and MetaWorld environments demonstrate the proposed approach consistently outperforms all baseline methods, suggesting that hierarchical prompting for decision transformers is an effective strategy to enable few-shot policy generalization.

Figures

Figures reproduced from arXiv: 2412.00979 by the authors.

Figure 1
Figure 1. The overall framework of the proposed HPDT. The top panel shows a demonstration trajectory from [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Meta-testing average performance of HPDT against baselines run over three random seeds facing unseen tasks. The [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Ablation studies on CHEETAH-VEL, CHEETAH-DIR, and PICK&PLACE. In (a)(b)(c), we compare each ablation with the full model on CHEETAH-DIR. Test tasks include running forward and backward. We show the accumulated reward for each task. The solid lines represent the full model HPDT for both tasks. The dashed lines represent the result of each ablation version. For CHEETAH-DIR, the global token is more important. In (d)(e… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Global tokens from different tasks 𝑔 𝑧 M𝑖 are well isolated and clustered from the same task. 100 50 0 50 100 100 50 0 50 100 (a) C-Vel: 35 tasks. 100 50 0 50 100 100 75 50 25 0 25 50 75 100 (b) A-Dir: 45 tasks [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 34 canonical work pages

  1. [1]

    Andrew G Barto and Sridhar Mahadevan. 2003. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems 13, 1-2 (2003), 41–77

    work page 2003

  2. [2]

    Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. 2019. A meta- transfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv:1901.10912 (2019)

    arXiv 2019

  3. [3]

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Ruther- ford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bog- dan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In ICML. PMLR, 2206–2240

    work page 2022

  4. [4]

    David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. 2022. When does return-conditioned supervised learning work for offline reinforcement learning? NeurIPS 35 (2022), 1542–1553

    work page 2022

  5. [5]

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling. NeurIPS 34 (2021), 15084–15097

    work page 2021

  6. [6]

    Alexandre

    André Correia and Luis A. Alexandre. 2023. Hierarchical Decision Transformer. In 2023 IEEE/RSJ IROS. 1661–1666. https://doi.org/10.1109/IROS55552.2023. 10342230

    arXiv 2023

  7. [7]

    Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. 2019. Search on the replay buffer: Bridging planning and reinforcement learning. NeurIPS 32 (2019)

    work page 2019

  8. [8]

    Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. 2021. Generalized de- cision transformer for offline hindsight information matching. arXiv preprint arXiv:2111.10364 (2021)

    arXiv 2021

Show all 62 references
  1. [9]

    Anirudh Goyal, Abram Friesen, Andrea Banino, Theophane Weber, Nan Rosemary Ke, Adria Puigdomenech Badia, Arthur Guez, Mehdi Mirza, Peter C Humphreys, Ksenia Konyushova, et al. 2022. Retrieval-augmented reinforcement learning. In ICML. PMLR, 7740–7765

    2022

  2. [10]

    Jake Grigsby, Zhe Wang, Nam Nguyen, and Yanjun Qi. 2021. Long-range trans- formers for dynamic spatiotemporal forecasting. arXiv preprint arXiv:2109.12218 (2021)

    2021 arXiv

  3. [11]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang

  4. [12]

    Bernhard Hengst. 2010. Hierarchical Reinforcement Learning

    2010

  5. [13]

    Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. 2023. Prompt-Tuning Decision Transformer with Preference Ranking. arXiv preprint arXiv:2305.09648 (2023)

    2023 arXiv

  6. [14]

    Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282 (2020)

    2020 arXiv

  7. [15]

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299 (2022)

    2022 arXiv

  8. [16]

    Michael Janner, Qiyang Li, and Sergey Levine. 2021. Offline Reinforcement Learning as One Big Sequence Modeling Problem. In NeurIPS

    2021

  9. [17]

    Daniel Kahneman. 2011. Thinking, fast and slow. macmillan

    2011

  10. [18]

    Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. 2019. Time2vec: Learning a vector representation of time. arXiv preprint arXiv:1907.05321 (2019)

    2019 arXiv

  11. [19]

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172 (2019)

    2019 arXiv

  12. [20]

    Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. 2022. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05618 (2022)

    2022 arXiv

  13. [21]

    Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, et al. 2022. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215 (2022)

    2022 arXiv

  14. [22]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In EMNLP, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). ACL, Online and Punta Cana, Dominican Republic

    2021

  15. [23]

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020)

    2020 arXiv

  16. [24]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al

  17. [25]

    Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer. In North American Asso- ciation for Computational Linguistics (NAACL). https://nlp.stanford.edu/pubs/ li2018transfer.pdf

    2018

  18. [26]

    Retrieval-augmented generation for knowledge-intensive nlp tasks.NeurIPS 33 (2020), 9459–9474

    2020

  19. [27]

    Licong Lin, Yu Bai, and Song Mei. 2023. Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. arXiv preprint arXiv:2310.08566 (2023)

    2023 arXiv

  20. [28]

    Lanqing Li, Rui Yang, and Dijun Luo. 2021. FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regu- larization. In ICRL. https://openreview.net/forum?id=8cpHIfgY4Dj

    2021

  21. [29]

    Tsendsuren Munkhdalai and Hong Yu. 2017. Meta networks. In ICML. PMLR, 2554–2563

    2017

  22. [30]

    Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn

  23. [31]

    Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang

  24. [32]

    Keiran Paster, Sheila Mcilraith, and Jimmy Ba. 2022. You can’t count on luck: Why decision transformers and rvs fail in stochastic environments. NeurIPS 35 (2022), 38966–38979

    2022

  25. [33]

    Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. 2018. Data- efficient hierarchical reinforcement learning. NeurIPS 31 (2018)

    2018

  26. [34]

    Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini

  27. [35]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9

    2019

  28. [36]

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexan- der Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al . 2022. A generalist agent. arXiv preprint arXiv:2205.06175 (2022)

    2022 arXiv

  29. [37]

    Hao Peng, Ankur P Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das

  30. [38]

    Kajetan Schweighofer, Markus Hofmarcher, Marius-Constantin Dinu, Philipp Renz, Angela Bitto-Nemling, Vihang Prakash Patil, and Sepp Hochreiter. 2021. Understanding the effects of dataset characteristics on offline reinforcement learn- ing. In Deep RL Workshop NeurIPS 2021

    2021

  31. [39]

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2024. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems 36 (2024)

    2024

  32. [40]

    IEEE Transactions on Neural Networks and Learning Systems (2023)

    A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems (2023)

    2023

  33. [41]

    Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An intro- duction. MIT press

    2018

  34. [42]

    Zhiwei Tang, Dmitry Rybin, and Tsung-Hui Chang. 2023. Zeroth-Order Opti- mization Meets Human Feedback: Provable Learning via Ranking Oracles. arXiv preprint arXiv:2303.03751 (2023)

    2023 arXiv

  35. [43]

    Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel

  36. [44]

    Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. 2017. Feudal networks for hierarchical reinforcement learning. In ICML. PMLR, 3540–3549

    2017

  37. [45]

    Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. 2022. Bootstrapped transformer for offline reinforcement learning. NeurIPS 35 (2022), 34748–34761

    2022

  38. [46]

    Yiqi Wang, Mengdi Xu, Laixi Shi, and Yuejie Chi. 2023. A Trajectory is Worth Three Sentences: Multimodal Transformer for Offline Reinforcement Learning. In UAI. https://openreview.net/forum?id=yE1_GpmDOPL

    2023

  39. [47]

    Shagun Sodhani, Amy Zhang, and Joelle Pineau. 2021. Multi-task reinforcement learning with context-based representations. In ICML. PMLR, 9767–9779

    2021

  40. [48]

    Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenen- baum, and Chuang Gan. 2022. Prompting Decision Transformer for Few-Shot Policy Generalization. In ICML. PMLR, 24631–24645

    2022

  41. [49]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022)

    2022 arXiv

  42. [50]

    Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ IROS . IEEE, 5026–5033. https: //doi.org/10.1109/IROS.2012.6386109

    2012

  43. [51]

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. 2020. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning . PMLR, 1094–1100

    2020

  44. [52]

    Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. 2017. Deep sets. NeurIPS 30 (2017)

    2017

  45. [53]

    Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau. 2020. Learn- ing robust state abstractions for hidden-parameter block MDPs. arXiv preprint arXiv:2007.07206 (2020)

    2020 arXiv

  46. [54]

    Zhe Wang, Jake Grigsby, Arshdeep Sekhon, and Yanjun Qi. 2022. ST-MAML: A Stochastic-Task based Method for Task-Heterogeneous Meta-Learning. InUAI. https://openreview.net/forum?id=rrlMyPUs9gc

    2022

  47. [55]

    Guangxiang Zhu, Zichuan Lin, Guangwen Yang, and Chongjie Zhang. 2019. Episodic reinforcement learning with associative memory. In ICLR

    2019

  48. [57]

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. NeurIPS 33 (2020), 5824–5836. WWW ’25 Companion, April 28 - May 02, 2025, Sydney , Australia Wang et al

    2020

  49. [61]

    Mingyang Zhou, Licheng Yu, Amanpreet Singh, Mengjiao Wang, Zhou Yu, and Ning Zhang. 2022. Unsupervised vision-and-language pre-training via retrieval- based multi-granular alignment. In CVPR. 16485–16494

    2022

  50. [2018]

    arXiv preprint arXiv:1810.06784 (2018)

    Promp: Proximal meta-policy search. arXiv preprint arXiv:1810.06784 (2018)

    2018 arXiv

  51. [2019]

    arXiv preprint arXiv:1904.04428 (2019)

    Text generation with exemplar-based adaptive decoding. arXiv preprint arXiv:1904.04428 (2019)

    2019 arXiv

  52. [2020]

    Retrieval augmented language model pre-training. In ICML. PMLR, 3929– 3938

  53. [2021]

    Offline meta-reinforcement learning with advantage weighting. In ICML. PMLR, 7780–7791

  54. [2023]

    In International Conference on Machine Learning

    Metadiffuser: Diffusion model as conditional planner for offline meta-rl. In International Conference on Machine Learning. PMLR, 26087–26105

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.