{"id":"923340ac-e0ed-4407-9391-f126595751fe","arxiv_id":"2412.00979","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HPDT improves few-shot policy generalization in offline meta-RL by adding a global task-level prompt and retrieval-based adaptive prompts to a decision transformer.","lead":"This paper introduces HPDT, a decision transformer variant that uses two levels of learned prompts, a global task token and per-timestep retrieved tokens, to generalize to new reinforcement learning tasks from a few demonstrations. It reports improved few-shot performance over prior decision transformer prompting methods on six of seven MuJoCo and MetaWorld benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HPDT's reported gains are confounded with the switch from lookup-table to Time2Vec time encoding; the paper's own HPDT wo T ablation underperforms PDT on Pick&Place, so the hierarchical-prompting effect is not established.","rationale":"The reader's weakest-assumption concern is that the global token in Eq. (4) is a mean over transitions from one sampled trajectory and may not be a sufficient statistic for task identity. That is a legitimate limitation, but it is about a single component's robustness. I find a more immediate, internally checkable threat to the central claim. The full HPDT package differs from PDT in two major ways: the hierarchical prompting mechanism and the time embedding. Table 4 provides HPDT wo T, which holds the time embedding fixed at PDT's lookup-table choice and still includes both global and adaptive tokens. Comparing this row with PDT in Table 3 directly isolates the hierarchical mechanism. On Pick&Place the mechanism is harmful (432.5 vs 497.5); on Cheetah-Dir and Cheetah-Vel the isolated gains are small (+8.4 and +6.6). The large headline gains, especially on Pick&Place (+72.0), coincide with the simultaneous introduction of Time2Vec, whose isolated contribution (HPDT vs HPDT wo T) is +137.0 on Pick&Place. The paper never runs PDT with Time2Vec, so the 'hierarchical prompting is effective' conclusion rests on a confounded comparison. This is an internal-consistency issue arising from the authors' own tables, not a disagreement with external consensus. It does not force rejection: the full method may still be a useful engineering contribution, and a controlled PDT+T2V experiment could vindicate the hierarchy on other environments. But as presented, the central mechanism claim is unsupported, which is exactly the kind of condition that the CONDITIONAL verdict should require. I therefore keep the verdict at CONDITIONAL and recommend the controlled time-embedding comparison as the decisive next step.","tokens_in":17203,"tokens_out":10481,"duration_ms":98002,"concrete_test":"Run a controlled baseline PDT+T2V: adopt the PDT architecture exactly as described and replace only its lookup-table time embedding with the Time2Vec formulation of Eq. (9), keeping all other hyperparameters, training epochs, seeds, and evaluation protocol identical. Evaluate on the seven benchmarks. If PDT+T2V matches or exceeds HPDT on Pick&Place and substantially closes the gaps on Cheetah-Dir and Cheetah-Vel, then the hierarchical prompts are not the source of the reported gains. A complementary check is to re-run HPDT wo T on all seven environments and test whether its performance is statistically separable from, or worse than, PDT.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that hierarchical global-plus-adaptive prompting is an effective strategy. But the full HPDT method changes two things relative to PDT: it adds hierarchical prompts and it replaces PDT's lookup-table time embedding with Time2Vec. The paper's own ablation HPDT wo T (Section 4, Table 4) removes Time2Vec while keeping the hierarchical prompts, giving an apples-to-apples test of the hierarchical mechanism with PDT's time encoding. On Pick&Place, HPDT wo T scores 432.5 ± 19.0, whereas PDT scores 497.5 ± 34.8 (Table 3): the hierarchical tokens actually reduce performance by about 65 points when the time embedding is held fixed. On Cheetah-Dir and Cheetah-Vel, the isolated hierarchy gains are small (+8.4 and +6.6), while the full-method gains are +26.2 and +11.2; on Pick&Place the full gain is +72.0, almost all of which comes from Time2Vec. Thus the abstract's conclusion that 'hierarchical prompting ... is an effective strategy' is not supported by the experiments as reported: the headline improvements over PDT are largely or entirely attributable to an orthogonal time-encoding change. No significance testing is reported, and the paper does not compare PDT+T2V (time embedding changed without hierarchy), so the attribution to hierarchy is untested.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes the Hierarchical Prompt Decision Transformer (HPDT), a method for offline meta-reinforcement learning that augments a Decision Transformer with two levels of soft prompts: a global token obtained by mean-pooling projected transition tuples from a demonstration trajectory (Eq. 4), and adaptive tokens obtained by KNN retrieval of similar rtg-state-action tuples that are added to the rollout tokens (Eqs. 6-8). The paper also replaces the lookup-table time embedding used by PDT with Time2Vec (Eq. 9). Experiments on five MuJoCo and two MetaWorld benchmarks compare HPDT with MACAW, PDT, PDT-FT, and PTDT, report average returns over three random seeds, and include ablations (HPDT w/o G, HPDT w/o A, HPDT w/o T) plus a hyperparameter robustness study. The central claim is that hierarchical global-plus-adaptive prompting consistently outperforms all baselines and is an effective few-shot generalization strategy.","tokens_in":17470,"tokens_out":5263,"duration_ms":48325,"significance":"If the causal contribution of hierarchical prompting were cleanly established, the method would be a useful addition to the OMRL literature: the global-token design is conceptually simple, the adaptive retrieval mechanism is plausible, and the paper includes useful ablations, a robustness table, and an inference-cost analysis. The evaluation uses standard held-out benchmarks and I see no circularity: test-task rollouts are not used to fit parameters. However, the headline claim of 'consistently outperforms all baseline methods' is not supported by the paper's own tables, and the attribution of the empirical gains to hierarchical prompting rather than to the simultaneous switch to Time2Vec is not established by the reported experiments. This is the load-bearing weakness of the paper.","major_comments":[{"comment":"The central causal claim that hierarchical prompting is effective is confounded with the time-encoding change. Full HPDT differs from PDT in two respects: it adds the hierarchical global/adaptive tokens and it replaces PDT's lookup-table time embedding with Time2Vec. The ablation HPDT w/o T keeps the hierarchical tokens while reverting to the lookup-table embedding. On Pick&Place, HPDT w/o T scores 432.5±19.0, which is below PDT's 497.5±34.8 in Table 3; on Cheetah-Vel and Cheetah-Dir the isolated hierarchy gains are only about +6.6 and +8.4. Thus the large reported gains over PDT, especially the +72.0 on Pick&Place, are largely or entirely attributable to Time2Vec rather than to the hierarchical prompting mechanism. To support the abstract's conclusion, the authors should report a PDT+Time2Vec baseline that changes only the time embedding while keeping PDT's static prompt, and should compare it against HPDT and HPDT w/o T.","section":"§4, Table 4 vs. Table 3"},{"comment":"The statement that HPDT 'consistently outperforms all baseline methods' is contradicted by Table 3. On Param-Walker, PTDT achieves 368.9±21.9 while HPDT achieves 367.3±32.5, so HPDT is not the best method there. On Reach, HPDT's 4832.2±5.2 is within one standard deviation of PTDT's 4830.5±2.9 and PDT-FT's 4828.3±6.5. The Section 4 claim of 'optimal results ... on six out of seven meta-environments' is accurate only if Param-Walker is the exception, but this should be stated explicitly in the abstract and conclusion rather than using 'consistently outperforms all baselines'. The claims should be tempered to match the evidence.","section":"Abstract and §4, Table 3"},{"comment":"The paper relies on only three random seeds and reports no significance tests or confidence intervals, yet several comparisons that feed the headline claims are within noise. For example, on Pick&Place, HPDT w/o G (568.0±5.5) and HPDT (569.5±5.1) are essentially tied, and on Reach the baselines overlap with HPDT. The authors should either provide more seeds, report paired significance tests, or explicitly identify which comparisons are robust. As it stands, the repeated wording 'significant improvements' and 'consistent improvements' is not supported by the reported statistics.","section":"§4, Tables 3-4, Figure 2"},{"comment":"The global token is defined as a mean over projected transition tuples, so the method implicitly assumes that the mean is a sufficient statistic for task identity. If tasks differ in higher-order statistics of the reward or transitions (e.g., variance or skew rather than mean), the mean-pooled global token cannot disambiguate them and the method would degrade toward the HPDT w/o G ablation. Figure 4 shows qualitative clustering but does not measure how much task-discriminative information survives pooling. I would ask the authors to quantify this, for example by reporting nearest-neighbor task classification accuracy on the global tokens, or by ablating the aggregator (mean vs. attention vs. max) on a task set where reward variance differs. This is a correctness-risk concern rather than a demonstrated failure, but it should be addressed or explicitly discussed as a limitation.","section":"§3, Eq. (4)"}],"minor_comments":[{"comment":"The sentence 'we randomly sample a demonstration trajectory Segment_z(O_M_i)' should refer to the demonstration set D_M_j of the new task, not the rollout set O_M_i; the notation is inconsistent with Section 2.","section":"§3, Evaluation paragraph"},{"comment":"Reference [13] is cited as PTDT and described as 'applied the zeroth-order optimization-based approach for prompt tuning', but the title 'Prompt-Tuning Decision Transformer with Preference Ranking' suggests the method is about preference ranking; if this is the intended PTDT baseline, the description should be aligned with the cited work, or the correct reference should be given.","section":"References"},{"comment":"The caption contains a duplicated label '3(d)(d)' in the second paragraph; it should read 'Figure 3(d)' and 'Figure 3(e)'.","section":"Figure 3 caption"},{"comment":"The text alternates between 'HPDT wo A' and 'HPDT w/o A' (similarly for G and T). Please use one consistent notation throughout the paper and figures.","section":"§4, Ablation paragraph"},{"comment":"For Pick&Place, the authors emphasize that adaptive tokens are important, but Table 4 shows HPDT w/o A (499.0±33.7) is essentially at the PDT level (497.5±34.8) rather than clearly below the full model; the discussion should acknowledge that the adaptive-token benefit on this environment is also partly entangled with the Time2Vec change.","section":"§4, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely a viable short conference contribution after a focused revision, but the current version overstates its evidence. The most important required change is to add the PDT+Time2Vec ablation and to revise the abstract and conclusions so that the causal claim about hierarchical prompting matches what the experiments can actually support. I do not see grounds for rejection, because the method is clearly specified and the ablations are informative, but the central attribution needs to be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: HPDT is a reasonable new combination—global mean-pooled token plus KNN-retrieved adaptive tokens, fused by addition, no fine-tuning—and the authors run the right first-pass ablations. But the abstract's claim that hierarchical prompting works is not supported by their own Table 4. The full method changes two things relative to PDT: hierarchy and time encoding. The HPDT wo T ablation keeps hierarchy but restores PDT's lookup-table time encoding. On Pick&Place that variant gets 432.5±19.0, below PDT's 497.5±34.8; on Cheetah-Dir and Cheetah-Vel the isolated hierarchy effect is small. So the large reported gains—especially Pick&Place, where full HPDT is 569.5 vs PDT 497.5—come mostly or entirely from Time2Vec. Without a PDT+T2V arm, the paper has not separated the two changes. That's the load-bearing flaw.\n\nWhat it does well: the method is described clearly; ablations for global vs adaptive tokens are sensible; Figure 4 shows global tokens do separate tasks on the environments shown; robustness to k and m' is a nice addition; and the baseline set includes MACAW, PDT, PDT-FT, and PTDT. The work is honest enough to include the HPDT wo T ablation, which makes the confound visible.\n\nOther soft spots: the abstract overstates \"consistently outperforms all baseline methods\"—Param-Walker HPDT 367.3±32.5 vs PTDT 368.9±21.9, and Reach is within noise. Only three seeds are reported, with no significance testing. No code or data is provided. Related work mentions FOCAL and MetaDiffuser but does not compare against them. The global-token mean pooling is plausible but unexamined for tasks where reward or transition differences show up in variance or skew rather than mean; that is a minor concern next to the Time2Vec confound.\n\nBottom line: this deserves a serious referee, but only with a deconfounding experiment required: run PDT+T2V and HPDT without Time2Vec on all environments, and report more seeds or significance tests. If the hierarchy effect survives, it is a useful subfield contribution. As is, I would not cite the headline result.","headline":"HPDT is a sensible new combination, but the paper's own ablation shows the headline gains are mostly Time2Vec, not hierarchical prompting.","tokens_in":18034,"tokens_out":2167,"would_cite":false,"duration_ms":20470,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a hierarchical prompting scheme, with a global token summarizing task dynamics and adaptive tokens retrieved from demonstrations, lets a decision transformer generalize to new tasks from a few demonstrations and…","keywords":["few-shot policy generalization","offline meta-reinforcement learning","decision transformer","hierarchical prompting","retrieval augmentation","soft prompt tokens","in-context learning"],"falsifier":"Train on two task families whose per-timestep transition tuples have identical means but differ in, for example, reward variance or the shape of the next-state distribution, then test whether HPDT distinguishes them during few-shot rollout; if it cannot, the mean-pooled global token is the reason.","tokens_in":16960,"feed_emoji":"🤖","tokens_out":9412,"duration_ms":79960,"temperature":0.7,"pith_summary":"The paper is trying to establish that a decision transformer can generalize to a new reinforcement-learning task from a handful of demonstrations if it receives two levels of prompt: a global token that summarizes the task's transition dynamics and reward pattern, and adaptive tokens that are retrieved from demonstration segments at each timestep. Current prompting methods for decision transformers use randomly sampled, static prompt segments, so they give the same guidance everywhere and often fail to identify the task. HPDT replaces those static prompts with learned soft tokens and a KNN retriever, and reports that this scheme achieves the best comparison results on six of seven MuJoCo and MetaWorld benchmarks, with the global token mattering most when task identities are discrete and the adaptive tokens mattering most when task identities vary continuously. A reader should care because it suggests that in-context prompting, without gradient updates, can be a competitive route to few-shot policy generalization.","feed_headline":"Two-level prompts beat static prompts on 6 of 7 few-shot RL tasks","feed_subtitle":"A single global token plus retrieved clips gives per-step guidance and adapts to new tasks without fine-tuning.","key_machinery":"The central object is the two-tier soft prompt. The global token $g^z_{\\mathcal{M}_i}$ is a single vector formed by the mean over the demonstration trajectory of $\\sigma(h_{\\theta_g}([\\hat{r}^*_{i,t}, s^*_{i,t}, a^*_{i,t}, s^*_{i,t+1}, \\hat{r}^*_{i,t+1}]))$, a GELU-activated linear projection of each transition tuple; it is meant to encode the task's transition dynamics and reward function. The adaptive tokens are produced by, for each rollout timestep $t$, taking the $k$ demonstration tuples whose return-to-go/state pairs are nearest in Euclidean distance to the current pair, and averaging their projected return-to-go, state, and action tokens. These adaptive tokens are fused with the rollout tokens by addition so that no new modality is introduced into the sequence, and a Time2Vec embedding provides relative time information with far fewer parameters than a lookup table. The causal transformer is trained end-to-end on the augmented sequences with action MSE, so the prompt layers and time embedding are learned together with the policy.","core_discovery":"On its own terms, the paper's discovery is that task identity and timestep-specific context can be separated into two soft-prompt mechanisms and that this separation helps a decision transformer adapt in offline meta-RL. The global token is computed by averaging, over one sampled demonstration trajectory, the projected transition tuples (current remaining reward, state, action, next state, next remaining reward); this average is intended to be a permutation-invariant summary of the transition dynamics and reward function. The adaptive tokens are computed for each rollout timestep by retrieving the closest demonstration tuples to the current remaining-reward/state pair and averaging their projections. When these tokens are prepended or added into the transformer input, the resulting model, HPDT, is reported to outperform MACAW, PDT, PTDT, and PDT-FT on six of the seven evaluation environments, and the abstract states the broader claim that it consistently outperforms all baselines across all seven.","pith_inferences":["If the global-token mean is sufficient for task identity, the same two-tier prompting could be dropped into other sequence-conditioned policies, such as trajectory or diffusion planners, by projecting their transition tuples.","The Euclidean KNN retriever in raw projected space is likely not the only viable choice; a learned or differentiable retriever could improve adaptive tokens on high-dimensional tasks, and a testable comparison would show whether retrieval quality is the bottleneck.","The single global token from one sampled demonstration may be brittle when demonstrations are noisy or when reward distributions differ in higher moments; using multiple demonstration trajectories or a learned set aggregator would be a natural stress test."],"forward_implications":["HPDT's global token compresses task identity into one vector, so the prompt length no longer grows with the number of demonstration timesteps needed to convey transition dynamics and rewards.","Adaptive tokens provide additional per-timestep guidance that matters most when task identities are similar, such as continuously varying goal velocities or goal positions.","The method's in-context prompting outperforms full model fine-tuning (PDT-FT) in the reported benchmarks, suggesting gradient-free adaptation can be sufficient for few-shot offline meta-RL.","Performance stays comparable when the demonstration length shrinks or the KNN neighborhood size changes, indicating the method does not require careful tuning of those hyperparameters.","The Time2Vec time embedding accelerates early training convergence relative to lookup-table embeddings because it has fewer parameters and is adjacency-aware."],"supporting_citations":[{"why":"Supplies the PDT baseline and the static-prompt formulation that HPDT's hierarchical prompts replace.","marker":"[48]"},{"why":"Establishes the decision transformer sequence-generation formulation on which the method is built.","marker":"[5]"},{"why":"Defines the offline meta-RL problem and the MACAW baseline HPDT compares against.","marker":"[28]"},{"why":"Provides retrieval-augmented generation, the inspiration for retrieving adaptive demonstration tokens.","marker":"[24]"},{"why":"Provides Time2Vec, the parameter-efficient time embedding HPDT uses instead of a lookup table.","marker":"[18]"},{"why":"Supplies the Meta-World benchmark environments used for evaluation.","marker":"[51]"},{"why":"Supplies the MuJoCo simulator and environments used for evaluation.","marker":"[43]"},{"why":"Supports the use of mean aggregation as a permutation-invariant set operator for the global token.","marker":"[52]"}],"fun_headline_variants":["Global and adaptive prompts beat static ones on 6/7 RL tasks","Two-level prompt hierarchy adapts RL per timestep","Retrieval-guided prompts improve few-shot RL adaptation","Task and step prompts separated boost few-shot generalization","Hierarchical prompts beat static ones across RL benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method relies on the average of projected transition tuples from one demonstration trajectory capturing enough of a task's identity; if two tasks have the same average but differ in other aspects of their dynamics or rewards, the global token cannot tell them apart.","fun_headline_variants_meta":{"raw":{"variants":["Global and adaptive prompts beat static ones on 6/7 RL tasks","Two-level prompt hierarchy adapts RL per timestep","Retrieval-guided prompts improve few-shot RL adaptation","Task and step prompts separated boost few-shot generalization","Hierarchical prompts beat static ones across RL benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000736,"raw_usage":{"total_tokens":3267,"prompt_tokens":899,"completion_tokens":2368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":2289}},"tokens_in":515,"tokens_out":2368,"duration_ms":14546,"temperature":1.0,"reasoning_tokens":2289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:47:46.786922+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train on two task families whose per-timestep transition tuples have identical means but differ in, for example, reward variance or the shape of the next-state distribution, then test whether HPDT distinguishes them during few-shot rollout; if it cannot, the mean-pooled global token is the reason.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PDT baseline and the static-prompt formulation that HPDT's hierarchical prompts replace."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the decision transformer sequence-generation formulation on which the method is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the offline meta-RL problem and the MACAW baseline HPDT compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MuJoCo simulator and environments used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the use of mean aggregation as a permutation-invariant set operator for the global token."}],"review_version":1}