Pith. sign in

REVIEW 1 major objections 5 minor 5 cited by

VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers

T0 review · 1 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Scaling a vector-quantized action tokenizer on over 100 times more data—mostly synthetic—raises real-robot success by up to 30 percentage points and nearly triples inference speed.

desk verdict Useful empirical result on VQ action tokenizers, but the headline scaling and domain-gap claims are confounded and overstated. read the letter →

arxiv 2507.01016 v1 pith:QVM56BKU submitted 2025-07-01 cs.RO cs.CV

Yating Wang , Haoyi Zhu , Mingyu Liu , Jiange Yang , Hao-Shu Fang , Tong He This is my paper
classification cs.ROcs.CV
keywords vectorquantizationactiontokenizationvision-language-actionmodelsVQ-VAEsyntheticdatascalingrobotmanipulationlong-horizontaskssim-to-realtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the action tokenizer, not just the policy network, is a bottleneck in vision-language-action models, and that the tokenizer can be scaled cheaply by training on synthetic trajectories. It reports that a residual vector-quantized tokenizer trained on over 100 times more action data than previous tokenizers transfers to real robots with roughly the same success as one trained on real data, and that more synthetic data yields higher downstream success, up to 30 percentage points on real long-horizon tasks. If true, this matters because it gives a low-compute route to better robot control: scale a small tokenizer rather than a 7-billion-parameter policy.

What carries the argument

The load-bearing object is the convolutional residual VQ-VAE action tokenizer: a VQ-VAE with a 2D temporal convolutional encoder-decoder, residual vector quantization with Nq quantization stages, sinusoidal time embeddings, and learnable action-type embeddings. It compresses an action chunk into Nq discrete codebook indices, with each quantization layer assigned a distinct non-overlapping token-ID range so the language model can predict all layers without semantic collisions. Frozen after training, it replaces OpenVLA's per-dimension 256-bin discretization, turning single-step action prediction into chunk-level token prediction; that compression ratio of 5 is what the paper credits for faster inference and reduced error accumulation.

What would settle it

Train the same tokenizer architecture on synthetic trajectories only and evaluate it on a broad set of real robot tasks beyond the three reported here, then add a small amount of real data and measure the change; if adding real data produces a large success-rate jump, or if synthetic-only training reconstructs real trajectories poorly, the marginal-domain-gap claim is false. A simpler check is to compute VQ reconstruction error on held-out real trajectories versus held-out synthetic trajectories.

Watch

Extended reading notes

Core claim

The paper's central claim is that robot action trajectories have a surprisingly small sim-to-real domain gap, so a residual VQ-VAE action tokenizer can be scaled on a mixture of real data (Open X-Embodiment) and much larger simulated datasets (LIBERO, ManiSkill, RLBench), and the resulting frozen tokenizer improves downstream VLA policies zero-shot. On LIBERO-90 the VQ tokenizer trained on ManiSkill plus RLBench raises success from 73.53% to 80.98%; on real tasks, the best variant lifts average success from 23% to 46.25% and reaches 50% and 30% on two long-horizon tasks where the baseline scored 15% and nearly 0%. The paper also claims the tokenizer nearly triples inference frequency (11.84 Hz versus 4.16 Hz) and that VQ-VAE tokenizers show approximately linear scaling as synthetic trajectory volume grows.

Load-bearing premise

The claim depends on simulated and real action trajectories being similar enough that training mostly on synthetic data does not sacrifice real-world performance, a premise the paper supports with only three real tasks and 20 trials each.

Editorial extensions

If this is right

  • If the claim is right, the cheapest way to improve a VLA policy is to improve its tokenizer: a tokenizer trained on one A100 for a week can be frozen and reused across tasks.
  • Existing large simulators can be repurposed as action data factories, because the paper's claim implies that the marginal sim-to-real gap will not punish the resulting policies.
  • Long-horizon tasks should disproportionately benefit, since chunk-level VQ prediction cuts the number of autoregressive steps and thus the chance of compounding errors.
  • Inference speeds nearly three times faster would make the approach practical for real-time control loops.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If scaling is truly linear, then success should keep rising with more synthetic data until some ceiling; this is a testable prediction the paper itself does not establish.
  • The marginal-domain-gap claim is made on three real tasks with 20 trials each; a natural extension is to test contact-rich and deformable-object tasks, where visual and physical sim-to-real gaps are usually largest.
  • The comparison to autoregressive action chunking suggests the benefit may come partly from chunking, not only from VQ expressiveness; a continuous action-chunk baseline with the same chunk length would isolate that factor.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes VQ-VLA, a two-stage pipeline in which a convolutional residual VQ-VAE action tokenizer is trained on large-scale real and synthetic trajectory data, frozen, and then used to replace the per-dimension binning tokenizer of OpenVLA during LoRA fine-tuning. The central claims are that increasing the volume of synthetic action data for tokenizer training yields a 'linear scaling' improvement in downstream VLA success rates, that the synthetic-to-real domain gap for action trajectories is marginal, and that the resulting tokenizer accelerates inference while producing smoother, longer-horizon behavior. The authors evaluate on LIBERO-90 in simulation and on six real-world Franka tasks, reporting gains over the OpenVLA baseline and ablations on encoder architecture, embeddings, and action chunking.

Significance. If the scaling claim were established, this would be a useful contribution: a lightweight, frozen action tokenizer that transfers across simulation and real robots would make VLA fine-tuning cheaper and faster, and the two-stage design is clean and easy to build on. The paper has several strengths: it uses public benchmarks and datasets, provides an honest comparison of Conv versus MLP tokenizer encoders, includes an action-chunking ablation, and openly reports the VQM failure case rather than hiding it. However, the load-bearing scaling claim is not supported by the experiments as reported, and the real-world evidence is statistically thin. The paper would be a solid method contribution if the scaling narrative were either backed by controlled experiments or removed in favor of a more modest data-composition claim.

major comments (1)
  1. [§3.3, §4.2.2] The training protocol is internally inconsistent and under-specified. Section 3.3 and §4.1.1 describe only VQM and VQM+R, yet §4.2.2 refers to 'the pre-trained VQO and VQO+L in the simulation experiments,' although no simulation results for VQO or VQO+L appear anywhere in the paper. In addition, the tokenizer configuration is not reproducible from the text: the number of VQ stages Nq, codebook size per stage, time-embedding and action-type-embedding dimensions, convolutional kernel sizes, and learning rate are all omitted. Please clarify which tokenizers were trained, on which datasets, and provide the full hyperparameter setting.
minor comments (5)
  1. [§2, Table 2, §5] There are several typos and inconsistencies: 'Releated Works' in the Section 2 heading, 'LIEBRO-90' in the Table 2 caption, and 'Fially' in Section 5.
  2. [§5] The Limitations section says the tokenizer 'can be further extended to larger-scale simulated datasets, such as RLbench,' but RLBench was already used in the VQM+R tokenizer in §4.1.1; this statement should be reworded or removed.
  3. [§3.2, Eq. (1)] Equation (1) uses symbols n, d, and k that are not all defined; please define k (the latent dimensionality) explicitly and state the input/output shapes for the encoder and decoder.
  4. [Figure 3] Figure 3 is difficult to read: the bars have no numerical labels, no error bars, and the task names are heavily wrapped. Please include the underlying per-task numbers and confidence intervals in a table or in the figure.
  5. [Abstract] The abstract says the tokenizer is built on 'the largest-scale action trajectory dataset to date' and uses 'over 100 times more data than previous approaches,' but no comparison dataset or exact count is given; please provide a concrete basis for this claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the tokenizer is trained on external data and frozen before downstream evaluation, so the reported gains are independent empirical results rather than consequences of a fitted input.

full rationale

The paper's derivation chain is not circular. The action tokenizer is trained on external trajectory datasets (Open X-Embodiment, LIBERO, ManiSkill, RLBench, Section 3.3) and then frozen; downstream VLA policies are fine-tuned with LoRA on LIBERO-90 or real-world tasks (Sections 3.4, 4.1, 4.2), so downstream success rates are not fed back into tokenizer training or hyperparameter selection. The central scaling claim ('the VQ-VAE tokenizers exhibit linear scaling properties in improving VLA success rates', Introduction) is weakly supported: Table 2 compares VQM (ManiSkill only) with VQM+R (ManiSkill + RLBench), confounding data volume with dataset composition, and Section 4.1.3's 'ablation study using only ManiSkill data' is a two-point comparison rather than a scale sweep. Section 5 also states that the tokenizer 'can be further extended to larger-scale simulated datasets,' acknowledging that no true scale curve was measured. However, this is an evidence-quality or correctness concern, not circularity: the predictor is fit to reconstruction targets on the training datasets, not to the evaluation outcomes it is later compared against. The 'marginal synthetic/real domain gap' claim in Section 4.2.4 is checked by an out-of-domain transfer experiment (VQ_L trained on LIBERO only and evaluated on real tasks, Table 3), which is an independent external test rather than a self-referential fit. No load-bearing argument reduces to a self-citation; the cited VQ-VAE, OpenVLA, LIBERO, ManiSkill, and RLBench results are external and reproducible. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its claims rest on standard VQ-VAE machinery plus a few dataset and preprocessing assumptions, most importantly the assumed sim-real domain gap.

free parameters (5)
  • VQ loss weight lambda = 4
    Set in Eq. (1) to balance reconstruction, codebook, and commitment losses; no sweep is reported.
  • Number of quantization stages Nq = not reported
    Determines the token ID range and reconstruction fidelity; the paper does not state its value.
  • Codebook size per VQ stage = 256 (implied by token ID offsets)
    Token ranges [256*(i-1), 256*i-1] imply 256 codes per stage, but the choice is not justified.
  • Action compression ratio / chunk length K = 5
    Each VQ token represents five actions; used in both training and the inference speed comparison.
  • Action-type embedding dimension = not reported
    Learnable embedding for XYZ, Euler, and gripper channels; the dimension is chosen by hand and not disclosed.
assumptions (3)
  • domain assumption The domain gap between synthetic and real action trajectories is marginal
    Used to justify co-training the tokenizer on LIBERO and ManiSkill for real-world tasks; supported only by a 3-task, 20-trial comparison in Section 4.2.4.
  • domain assumption No-op actions are filtered from training data following OpenVLA
    Stated in Section 4.1.1; this preprocessing choice affects the action distribution the tokenizer learns.
  • standard math The residual VQ-VAE loss from Eq. (1) yields useful discrete codes
    Borrowed from SoundStream [48]; the stop-gradient and commitment terms are standard and not re-derived here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers." pith.science (2026). https://pith.science/paper/QVM56BKU

@misc{pith2026250701016,
  author       = {Pith},
  title        = {Pith review of: VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QVM56BKU}},
  note         = {Machine review of arXiv:2507.01016}
}
read the original abstract

In this paper, we introduce an innovative vector quantization based action tokenizer built upon the largest-scale action trajectory dataset to date, leveraging over 100 times more data than previous approaches. This extensive dataset enables our tokenizer to capture rich spatiotemporal dynamics, resulting in a model that not only accelerates inference but also generates smoother and more coherent action outputs. Once trained, the tokenizer can be seamlessly adapted to a wide range of downstream tasks in a zero-shot manner, from short-horizon reactive behaviors to long-horizon planning. A key finding of our work is that the domain gap between synthetic and real action trajectories is marginal, allowing us to effectively utilize a vast amount of synthetic data during training without compromising real-world performance. To validate our approach, we conducted extensive experiments in both simulated environments and on real robotic platforms. The results demonstrate that as the volume of synthetic trajectory data increases, the performance of our tokenizer on downstream tasks improves significantly-most notably, achieving up to a 30% higher success rate on two real-world tasks in long-horizon scenarios. These findings highlight the potential of our action tokenizer as a robust and scalable solution for real-time embodied intelligence systems, paving the way for more efficient and reliable robotic control in diverse application domains.Project website: https://xiaoxiao0406.github.io/vqvla.github.io

Figures

Figures reproduced from arXiv: 2507.01016 by the authors.

Figure 1
Figure 1. The VQ-VLA pipeline, consisting of two main stages: (1) training a general convolutional residual VQ-VAE and (2) fine-tuning OpenVLA using the LoRA approach. Specifically, a general convolutional residual VQ-VAE is first trained on the Open X-Embodiment dataset, LIEBRO, and ManiSkill datasets. The trained VQ-VAE is then frozen and serves as an action tokenizer for OpenVLA, replacing the simple binning method. In the… view at source ↗
Figure 2
Figure 2. All Evaluation environments:We conduct comprehen￾sive evaluations of VQ-VLA in both simulation and real-world settings. In simulation, evaluations are performed on the LIBERO￾90 benchmark within the LIBERO dataset. And six diverse tasks are designed for real-world testing. 4.2.2. Performance on Short-Horizon Tasks Based on the pre-trained VQO and VQO+L in the simulation experiments, we further incorporated 120k synt… view at source ↗
Figure 3
Figure 3. Real-world experimental results: We compare the performance of Baseline, VQO, VQO+L, and VQO+L+M on both short-horizon and long-horizon tasks. In terms of the average success rate, all VQ-based models outperform the Baseline. The best-performing model, VQO+L+M, achieves a success rate that is 23.25% higher than the Baseline on both short-horizon and long-horizon tasks. Additionally, the results show that VQO+L+M out… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A self-supervised multimodal encoder trained with vision, proprioception, and force yields a vision-only latent that recovers end-effector state and force above vision baselines on RH20T, with modest absolute force accuracy.

  2. EDAR: Learning Environment-Dependent Action Representations for Robotic Manipulation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Action latents supervised by both control reconstruction and environment-conditioned visual consequences outperform trajectory-centric tokenizers for robotic manipulation, especially long-horizon tasks.

  3. PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies

    cs.CV 2026-02 unverdicted novelty 6.0 of 10

    Pretraining a vision-language model to output discrete 3D pose tokens on large non-robotic data, before training a robot action head, improves downstream manipulation success and data efficiency.

  4. Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-training

    cs.RO 2025-12 conditional novelty 6.0 of 10

    A 6K-question embodied-reasoning benchmark plus a flow-matching action tokenizer let one 3B vision-language model reason and manipulate better than continuous- or discrete-action VLA baselines.

  5. RynnVLA-002: A Unified Vision-Language-Action and World Model

    cs.RO 2025-11 conditional novelty 5.0 of 10

    A single model that jointly predicts robot actions and future images outperforms separate action-only and video-only models on LIBERO and real SO100 manipulation tasks.

Reference graph

Works this paper leans on

55 extracted references · 16 canonical work pages · cited by 5 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1

    arXiv 2023

  2. [2]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebo- tar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. 2

    arXiv 2022

  3. [3]

    Flamingo: A visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeffrey Donahue, Pauline Luc, Alexis Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: A visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 35:23716–23736,

    work page

  4. [4]

    Minivla: A better vla with a smaller footprint

    Suneel Belkhale and Dorsa Sadigh. Minivla: A better vla with a smaller footprint. https://ai.stanford.edu/blog/minivla/,

    work page

  5. [5]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 2

    arXiv 2022

  6. [6]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 1, 2

    arXiv 2023

  7. [7]

    Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation

    Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024. 2

    arXiv

  8. [8]

    Anyvlm: Unified vision-language model for any robot morphology

    Wei Chen, Yifeng Zhang, Hao Li, Xufeng Wang, and Ming Liu. Anyvlm: Unified vision-language model for any robot morphology. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 1–8. IEEE, 2024. 2

    work page 2024

Show all 55 references
  1. [9]

    Pali-x: On scaling up a multilingual vision and language model

    Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebas- tian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565, 2023. 2

    2023 arXiv

  2. [10]

    Igor: Image-goal representations are the atomic control units for foundation models in embodied ai

    Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Cathera Yang, Li Zhao, and Jiang Bian. Igor: Image-goal representations are the atomic control units for foundation models in embodied ai. arXiv preprint arXiv:2411.00785, 2024. 2

    2024 arXiv

  3. [11]

    Diffusion policy: Visuomotor policy learning via action diffu- sion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffu- sion. The International Journal of Robotics Research, page 02783649241273668, 2023. 2

    2023

  4. [12]

    Keypoint action to- kens enable in-context imitation learning in robotics

    Norman Di Palo and Edward Johns. Keypoint action to- kens enable in-context imitation learning in robotics. arXiv preprint arXiv:2403.19578, 2024. 2

    2024 arXiv

  5. [13]

    Palm-e: an embod- ied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: an embod- ied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, page...

    2023

  6. [14]

    Exploiting llm quantization

    Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, and Martin Vechev. Exploiting llm quantization. Advances in Neural Information Processing Systems , 37:41709–41732,

  7. [15]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 1, 2

    2021

  8. [16]

    Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot

    Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. arXiv preprint arXiv:2307.00595, 2023. 2

    2023 arXiv

  9. [17]

    A new algorithm for data compression

    Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2):23–38, 1994. 2

    1994

  10. [18]

    Act3d: 3d feature field transformers for multi-task robotic manipulation

    Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, and Ka- terina Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation. In 7th Annual Conference on Robot Learning. 2

  11. [19]

    Relay policy learning: Solving long- horizon tasks via imitation and reinforcement learning

    Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long- horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956, 2019. 2

    1910 arXiv

  12. [20]

    Deep reinforce- ment learning in parameterized action space

    Matthew Hausknecht and Peter Stone. Deep reinforce- ment learning in parameterized action space. arXiv preprint arXiv:1511.04143, 2015. 2

    2015 arXiv

  13. [21]

    Cogvideo: Large-scale pretraining for text-to-video gen- eration via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video gen- eration via transformers. arXiv preprint arXiv:2205.15868,

    arXiv

  14. [22]

    Rlbench: The robot learning benchmark & learning environment

    Stephen James, Zicong Ma, David Rovick Arrojo, and An- drew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020. 4, 8

    2020

  15. [23]

    Pyramidal flow matching for efficient video generative modeling

    Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954, 2024. 2

    2024

  16. [24]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. 1, 2, 4

    2024 arXiv

  17. [25]

    Fine-tuning vision-language-action models: Optimizing speed and suc- cess

    Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and suc- cess. arXiv preprint arXiv:2502.19645, 2025. 2

    2025 arXiv

  18. [26]

    Be- havior generation with latent actions

    Seungjae Lee, Yibin Wang, Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Be- havior generation with latent actions. arXiv preprint arXiv:2403.03181, 2024. 1, 2

    2024 arXiv

  19. [27]

    Libero: Benchmarking knowledge transfer for lifelong robot learning

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023. 1, 2, 3, 4

    2023 arXiv

  20. [28]

    Open-magvit2: An open-source project toward democratizing auto-regressive visual generation.arXiv preprint arXiv:2409.04410, 2024

    Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation.arXiv preprint arXiv:2409.04410, 2024. 2

    2024 arXiv

  21. [29]

    Language models are few-shot learners

    Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 1:3, 2020. 1

    2005 arXiv

  22. [30]

    Quest: Self-supervised skill abstractions for learning continuous control

    Atharva Mete, Haotian Xue, Albert Wilcox, Yongxin Chen, and Animesh Garg. Quest: Self-supervised skill abstractions for learning continuous control. Advances in Neural Informa- tion Processing Systems, 37:4062–4089, 2025. 2

    2025

  23. [31]

    Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations

    Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xu- anlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. arXiv preprint arXiv:2107.14483,

    arXiv

  24. [32]

    Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0

    Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Ab- hishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Poo- ley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE Inte...

    2024

  25. [33]

    Fast: Efficient action tokenization for vision- language-action models

    Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision- language-action models. arXiv preprint arXiv:2501.09747,

    arXiv

  26. [34]

    Language models are unsuper- vised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsuper- vised multitask learners. OpenAI blog, 1(8):9, 2019. 2

    2019

  27. [35]

    Taming scalable visual tok- enizer for autoregressive image generation

    Fengyuan Shi, Zhuoyan Luo, Yixiao Ge, Yujiu Yang, Ying Shan, and Limin Wang. Taming scalable visual tok- enizer for autoregressive image generation. arXiv preprint arXiv:2412.02692, 2024. 2

    2024 arXiv

  28. [36]

    Llm pruning and distillation in practice: The minitron approach

    Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng, Zijia Chen, Yoshi Suhara, Shizhe Diao, et al. Llm pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408....

    2024 arXiv

  29. [37]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 1, 2

    2024 arXiv

  30. [38]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024. 2

    2024 arXiv

  31. [39]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 2

    2017

  32. [40]

    Chatgpt for robotics: Design principles and model abilities

    Sai H Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Ieee Access, 2024. 2

    2024

  33. [41]

    Any-point trajectory modeling for policy learning

    Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023. 2

    2023 arXiv

  34. [42]

    Tinyvla: Towards fast, data-efficient vision- language-action models for robotic manipulation

    Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, et al. Tinyvla: Towards fast, data-efficient vision- language-action models for robotic manipulation. arXiv preprint arXiv:2409.12514, 2024. 2

    2024 arXiv

  35. [43]

    Transferring foundation models for generalizable robotic ma- nipulation

    Jiange Yang, Wenhui Tan, Chuhao Jin, Keling Yao, Bei Liu, Jianlong Fu, Ruihua Song, Gangshan Wu, and Limin Wang. Transferring foundation models for generalizable robotic ma- nipulation. arXiv preprint arXiv:2306.05716, 2023. 2

    2023 arXiv

  36. [44]

    Spatiotemporal predictive pre-training for robotic motor control

    Jiange Yang, Bei Liu, Jianlong Fu, Bocheng Pan, Gangshan Wu, and Limin Wang. Spatiotemporal predictive pre-training for robotic motor control. arXiv preprint arXiv:2403.05304,

    arXiv

  37. [45]

    Tra-moe: Learning trajectory prediction model from multiple domains for adaptive policy condition- ing

    Jiange Yang, Haoyi Zhu, Yating Wang, Gangshan Wu, Tong He, and Limin Wang. Tra-moe: Learning trajectory prediction model from multiple domains for adaptive policy condition- ing. arXiv preprint arXiv:2411.14519, 2024. 2

    2024 arXiv

  38. [46]

    Latent action pretraining from videos

    Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. arXiv preprint arXiv:2410.11758, 2024. 2

    2024 arXiv

  39. [47]

    Robotic control via embodied chain-of-thought reasoning

    Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning. arXiv preprint arXiv:2407.08693, 2024. 2

    2024 arXiv

  40. [48]

    Soundstream: An end-to- end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to- end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021. 1, 2, 3

    2021

  41. [49]

    Moviedreamer: Hierarchical generation for coherent long visual sequence

    Canyu Zhao, Mingyu Liu, Wen Wang, Weihua Chen, Fan Wang, Hao Chen, Bo Zhang, and Chunhua Shen. Moviedreamer: Hierarchical generation for coherent long visual sequence. arXiv preprint arXiv:2407.16655, 2024. 2

    2024

  42. [50]

    Diception: A generalist diffusion model for visual perceptual tasks

    Canyu Zhao, Mingyu Liu, Huanyi Zheng, Muzhi Zhu, Zhiyue Zhao, Hao Chen, Tong He, and Chunhua Shen. Diception: A generalist diffusion model for visual perceptual tasks. arXiv preprint arXiv:2502.17157, 2025. 2

    2025

  43. [51]

    Learning fine-grained bimanual manipulation with low- cost hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low- cost hardware. arXiv preprint arXiv:2304.13705, 2023. 2

    2023 arXiv

  44. [52]

    Vlas: Vision-language-action model with speech instructions for customized robot manipulation

    Wei Zhao, Pengxiang Ding, Min Zhang, Zhefei Gong, Shuanghao Bai, Han Zhao, and Donglin Wang. Vlas: Vision-language-action model with speech instructions for customized robot manipulation. arXiv preprint arXiv:2502.13508, 2025. 2

    2025 arXiv

  45. [53]

    3d-vla: A 3d vision-language-action generative world model

    Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model. arXiv preprint arXiv:2403.09631, 2024. 2

    2024 arXiv

  46. [54]

    Point cloud matters: Rethinking the impact of different observation spaces on robot learn- ing

    Haoyi Zhu, Yating Wang, Di Huang, Weicai Ye, Wanli Ouyang, and Tong He. Point cloud matters: Rethinking the impact of different observation spaces on robot learn- ing. Advances in Neural Information Processing Systems, 37: 77799–77830, 2024. 2

    2024

  47. [55]

    Spa: 3d spatial-awareness enables effective embodied representation

    Haoyi Zhu, Honghui Yang, Yating Wang, Jiange Yang, Limin Wang, and Tong He. Spa: 3d spatial-awareness enables effective embodied representation. arXiv preprint arXiv:2410.08208, 2024. 2

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.