Pith. sign in

REVIEW 2 cited by

Compositional Foundation Models for Hierarchical Planning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.08587 v2 pith:CWSX7FNM submitted 2023-09-15 cs.LG cs.AIcs.RO

Anurag Ajay , Seungwook Han , Yilun Du , Shuang Li , Abhi Gupta , Tommi Jaakkola , Josh Tenenbaum , Leslie Kaelbling
show 2 more authors
Akash Srivastava Pulkit Agrawal
This is my paper · ORCID
classification cs.LGcs.AIcs.RO
keywords modelfoundationhierarchicallong-horizonmodelsplanningplansreasoning
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

To make effective decisions in novel environments with long-horizon goals, it is crucial to engage in hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the underlying plans, and executing actions in accordance with the devised plan through visual-motor control. We propose Compositional Foundation Models for Hierarchical Planning (HiP), a foundation model which leverages multiple expert foundation model trained on language, vision and action data individually jointly together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement. We illustrate the efficacy and adaptability of our approach in three different long-horizon table-top manipulation tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning

    cs.LG 2024-12 conditional novelty 6.0 of 10

    MoDE, a mixture-of-experts diffusion transformer with noise-conditioned routing, reports state-of-the-art results on CALVIN and LIBERO with lower inference FLOPs than dense baselines.

  2. Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Task Decodability, a k-NN measure of how separable a task is in a model's middle-layer representations, tracks and predicts in-context learning accuracy, and early-layer finetuning improves it more than late-layer finetuning.

Pith tools