Pith. sign in

REVIEW 6 cited by

Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.19578 v3 pith:GADUKDWM submitted 2024-03-28 cs.RO cs.LGcs.NE

Norman Di Palo , Edward Johns This is my paper
classification cs.ROcs.LGcs.NE
keywords actionimitationlearningvisualkeypointlanguageobservationstokens
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We show that off-the-shelf text-based Transformers, with no additional training, can perform few-shot in-context visual imitation learning, mapping visual observations to action sequences that emulate the demonstrator's behaviour. We achieve this by transforming visual observations (inputs) and trajectories of actions (outputs) into sequences of tokens that a text-pretrained Transformer (GPT-4 Turbo) can ingest and generate, via a framework we call Keypoint Action Tokens (KAT). Despite being trained only on language, we show that these Transformers excel at translating tokenised visual keypoint observations into action trajectories, performing on par or better than state-of-the-art imitation learning (diffusion policies) in the low-data regime on a suite of real-world, everyday tasks. Rather than operating in the language domain as is typical, KAT leverages text-based Transformers to operate in the vision and action domains to learn general patterns in demonstration data for highly efficient imitation learning, indicating promising new avenues for repurposing natural language models for embodied tasks. Videos are available at https://www.robot-learning.uk/keypoint-action-tokens.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos

    cs.RO 2025-09 conditional novelty 7.0 of 10

    Trained only on unlabeled human play videos, MimicDroid lets a GR1 humanoid perform new manipulation tasks from one to three demonstration videos, with roughly twice the real-world success of prior video-conditioned methods.

  2. Try Once, Then Optimal: De-Redundified Procedure Memory for Cross-Episode Exploration Amortization

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Object-centric procedure memory amortizes hidden-state exploration across encounters, cutting robot manipulation operations 16–30% at non-regressing success.

  3. RoboSSM: Scalable In-context Imitation Learning via State-Space Models

    cs.RO 2025-09 conditional novelty 6.0 of 10

    RoboSSM shows that a state-space model backbone can extend in-context imitation learning to prompts much longer than those seen in training, where a Transformer-based baseline degrades.

  4. Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions

    cs.RO 2025-06 conditional novelty 6.0 of 10

    A semantic keypoint graph matched to novel objects lets imitation-learned manipulation policies generalize with a quarter of the demonstrations.

  5. VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers

    cs.RO 2025-07 conditional novelty 5.0 of 10

    A convolutional residual VQ-VAE action tokenizer trained on over 100x more data than prior work improves OpenVLA success rates and inference speed on several manipulation tasks.

  6. CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity

    cs.RO 2025-06

Pith tools