CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning

Chunhe Xia; Dongchi Huang; Lin Zhao; Tianle Zhang; Yihang Li; Zhirui Fang

REVIEW 3 major objections 3 minor 5 cited by

CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CO-RFT, an offline RL algorithm that treats action chunks as decision units, claims to beat supervised fine-tuning of vision-language-action models: 57% higher success rate, 22.3% shorter cycle time, and 44.3% success in unseen positions.

desk verdict Plausible VLA fine-tuning recipe with strong empirical claims, but the chunked TD target needs precise definition and the reported numbers need error bars before I'd trust them. read the letter →

arxiv 2508.02219 v1 pith:BXRGR236 submitted 2025-08-04 cs.RO cs.LG

Dongchi Huang , Zhirui Fang , Tianle Zhang , Yihang Li , Lin Zhao , Chunhe Xia This is my paper

classification cs.ROcs.LG

keywords vision-language-actionmodelsofflinereinforcementlearningactionchunkingtemporaldifferenceroboticcontrolimitationfine-tuningsampleefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a vision-language-action (VLA) model can be fine-tuned with offline reinforcement learning from as few as 30 to 60 demonstrations, and that this beats supervised fine-tuning on real robot tasks. The authors argue that the key is to make the learning update mirror the model's own output format: VLA policies emit a chunk of actions at once, so the reinforcement-learning target should treat the whole chunk as one decision unit rather than bootstrapping after every individual action. If the claim holds, robot policies could be improved from small offline datasets without risky online interaction, with reported gains of 57% in success rate and 22.3% in cycle time, along with a 44.3% success rate in previously unseen positions.

What carries the argument

The machinery is Chunked RL, a framework that redefines the temporal difference update so that a chunk of $K$ future actions is treated as one decision unit by the value estimate, with the TD target bootstrapping across chunk boundaries rather than at every action step. Action chunking is the VLA model's convention of predicting a fixed-length sequence of actions in one forward pass. A policy is first initialized by full-parameter imitation learning, then trained offline with this chunked TD objective; the chunk is the object that carries the credit-assignment signal.

What would settle it

A reader could settle this by re-running the method with the chunked TD target replaced by a per-action TD target while keeping the imitation initialization, dataset, and reward signal identical; if the success-rate advantage disappears, the chunked formulation is not what drives the gains. A complementary check is to compare the learned Q-values against Monte Carlo returns on held-out trajectories: systematic over- or under-prediction inside chunks would indicate biased credit assignment.

Watch

Extended reading notes

Core claim

The central claim is that extending temporal difference (TD) learning to action chunks makes offline RL sample-efficient enough to fine-tune a vision-language-action model from a limited set of demonstrations, and that the resulting policy outperforms supervised fine-tuning in real-world environments. The paper reports that its algorithm, CO-RFT, improves success rate by 57% and reduces cycle time by 22.3% relative to previous supervised methods, and that it reaches a 44.3% success rate in positions not seen during training. The procedure is two-stage: first full-parameter imitation learning initializes both the backbone and the policy, then offline RL with action chunking optimizes the initialized policy.

Load-bearing premise

The load-bearing assumption is that temporal difference learning stays valid as a credit-assignment signal when actions are grouped into chunks, so that rewards inside a chunk can be attributed to the entire chunk instead of to individual actions.

Editorial extensions

If this is right

Fine-tuning VLA models from very small offline datasets is feasible when the RL update respects the chunk structure of the policy.
Offline RL can improve both success rate and speed of a real-world robot policy compared with supervised fine-tuning, without online exploration.
The two-stage recipe of imitation initialization followed by chunked offline RL provides a practical template for adapting general VLA models to specific tasks.
Positional generalization to unseen configurations can be improved by the chunked RL objective, not just by more data or larger models.
Chunk size becomes a design parameter of the learning objective itself, not just of the policy output format.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the chunked TD extension is correct, action chunking is not merely an inference-time convenience but a structural inductive bias for credit assignment; that would make chunk size worth tuning jointly with the discount factor.
The same chunked TD idea may transfer to other sequence-level policy representations, such as diffusion policies or trajectory transformers, which also emit multi-step actions.
A crisp comparison against per-step TD on the same data would reveal whether the benefit is sample efficiency (a gap that closes with more data) or a genuine asymptotic advantage.
The reported 44.3% unseen-position success rate suggests the chunked objective regularizes the policy, but without calibration or uncertainty estimates it remains unclear how far outside the training distribution the method will stay reliable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's my take on CO-RFT.

The one-two punch: it's a plausible and timely recipe for VLA fine-tuning—full-param IL init, then offline RL on action chunks, trained on 30–60 demos—and the reported gains (57% success, 22.3% cycle time) are large enough to take seriously. But the central theoretical move, "extend TD to action chunking," is underspecified, and the copy I have is corrupted so I couldn't check the target equation. The empirical claims are also uncheckable: no error bars, no ablations, no baselines beyond "previous supervised methods."

What's genuinely new: applying chunked offline RL to VLA models is a reasonable, practical combination that hasn't been the focus before, and the chunked TD extension is the piece to probe. If it works, it's useful. Credit where due: the experimental setup—real-world, 30–60 demos, unseen-position generalization—is exactly the bottleneck people care about.

The soft spots: the stress-test concern is real. A TD target over a chunk is unbiased only if the policy actually executes the whole chunk open-loop. If they use receding-horizon execution (predict H, execute one, re-predict), the later chunk actions are never the actions taken at the states the target bootstraps on, so credit is misassigned. The abstract doesn't say which protocol they use. That's not a fatal flaw—open-loop execution is a well-defined semi-MDP—but the authors need to state it and give the target equation. Also, with only 30–60 demos, the 57% improvement could easily come from the IL initialization or the chunking representation rather than the RL objective; the paper needs ablations separating those. And "novel framework" is overclaiming; this is an extension of existing offline RL to chunked action spaces.

Who this is for: robotics RL folks working on VLA fine-tuning, and anyone implementing efficient post-IL RL. A serious referee should insist on the execution protocol, the exact Bellman target, error bars across seeds, and a comparison against IL init alone and chunked IL alone. The idea deserves a proper hearing, so I'd send it to review rather than desk reject.

Referee Report

3 major / 3 minor

Summary. The paper proposes CO-RFT, a two-stage fine-tuning method for vision-language-action (VLA) models: first imitation learning with full-parameter fine-tuning, then offline reinforcement learning with action chunking. The abstract reports a 57% improvement in success rate, a 22.3% reduction in cycle time over supervised baselines, and a 44.3% success rate in previously unseen positions. The submitted full text is corrupted and unreadable, so the method description, equations, experiments, and baseline definitions are not accessible for review.

Significance. If substantiated, the claimed combination of offline RL with action chunking for fine-tuning VLA models from only 30 to 60 demonstrations would be a practically valuable contribution. However, the current submission provides no verifiable methodological detail, no baseline definitions, no error bars, and no task descriptions. The central contribution, the chunked temporal-difference (TD) extension, cannot be checked. The reported gains could plausibly arise from the imitation-learning initialization, the chunking representation, reward weighting, or favorable task selection rather than from a correct chunked RL objective. The significance of the paper cannot be assessed in its present form.

major comments (3)

[Full text (all sections after the abstract)] The manuscript is supplied in a corrupted, unreadable encoding; none of the equations, algorithm descriptions, experimental setup, or baseline definitions can be parsed. The central claim and the proposed 'Chunked RL' framework are therefore unverifiable. Please resubmit a readable PDF and ensure the technical content is complete.
[Abstract (claims of 57% improvement and 22.3% cycle-time reduction)] These numbers are stated without any definition of the baseline methods, number of trials, variance, or statistical significance. As a result, it is impossible to tell whether the improvement is robust across tasks or an artifact of a few favorable trials. Please report per-task results, standard deviations, and explicit baseline configurations.
[Method (Chunked RL / TD extension, not visible due to corruption)] The novelty rests on extending TD learning to action chunks. The validity of such an extension depends critically on the execution protocol: open-loop execution yields a semi-MDP with a target of the form R + gamma^H V, while receding-horizon execution makes the chunked target biased toward a policy that is never executed. The submission must state which protocol is used and provide the exact target equation. In the current corrupted text, the loss symbols are unreadable, so this load-bearing point cannot be assessed.

minor comments (3)

[Abstract] The acronym 'CO-RFT' is not defined in the abstract; please spell out 'Chunked Offline Reinforcement Fine-Tuning' at first use.
[Abstract] The phrase '30 to 60 samples' is ambiguous: are these trajectories, demonstrations, or training episodes? Please clarify in the experimental section.
[Abstract] The term 'cycle time' is not defined; please specify whether it is wall-clock time per episode, number of control steps, or another metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity is evident: the abstract's RL procedure and empirical comparisons are not defined in terms of the reported outcomes.

full rationale

The derivable content provided (abstract and a corrupted full-text rendering) does not exhibit any step in which an output is defined by the quantity it claims to predict. The method (1) initializes with imitation learning, (2) applies offline RL with action chunking, and (3) reports success-rate and cycle-time measurements against previous supervised methods; these are external, falsifiable comparisons rather than quantities reconstructed from the method's own definitions. The phrase 'we extend temporal difference (TD) learning to incorporate action chunking' is a methodological claim, not a definition of the evaluation metric, and no target equation is available to show that the chunked TD objective is equivalent to the measured improvement. No load-bearing self-citations are present in the visible text. Because the full text is corrupted, I could not inspect the chunked TD target equation, but per the rules a circularity claim requires quoting the paper and exhibiting the specific reduction; no such reduction is visible, so the honest finding is no significant circularity (score 0).

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No hyperparameters or fitted constants are identified because only the abstract is available. The central assumptions are that offline RL can improve on an imitation-learned initialization from very few demonstrations, and that TD learning can be extended to action chunks without loss of correctness. The action chunk size is a design choice that is not reported in the abstract, so no concrete free parameter can be listed.

assumptions (2)

domain assumption Offline RL from 30 to 60 demonstrations can improve a policy initialized by imitation learning.
The abstract asserts that after IL initialization, offline RL optimizes the policy; this assumes the demonstrations contain enough signal for RL to improve on the IL solution.
ad hoc to paper Temporal difference learning extends to action chunks with valid credit assignment.
The abstract announces this extension without derivation; it is a modeling assumption introduced specifically for this method and not standard in the RL literature.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/BXRGR236

@misc{pith2026250802219,
  author       = {Pith},
  title        = {Pith review of: CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BXRGR236}},
  note         = {Machine review of arXiv:2508.02219}
}

read the original abstract

Vision-Language-Action (VLA) models demonstrate significant potential for developing generalized policies in real-world robotic control. This progress inspires researchers to explore fine-tuning these models with Reinforcement Learning (RL). However, fine-tuning VLA models with RL still faces challenges related to sample efficiency, compatibility with action chunking, and training stability. To address these challenges, we explore the fine-tuning of VLA models through offline reinforcement learning incorporating action chunking. In this work, we propose Chunked RL, a novel reinforcement learning framework specifically designed for VLA models. Within this framework, we extend temporal difference (TD) learning to incorporate action chunking, a prominent characteristic of VLA models. Building upon this framework, we propose CO-RFT, an algorithm aimed at fine-tuning VLA models using a limited set of demonstrations (30 to 60 samples). Specifically, we first conduct imitation learning (IL) with full parameter fine-tuning to initialize both the backbone and the policy. Subsequently, we implement offline RL with action chunking to optimize the pretrained policy. Our empirical results in real-world environments demonstrate that CO-RFT outperforms previous supervised methods, achieving a 57% improvement in success rate and a 22.3% reduction in cycle time. Moreover, our method exhibits robust positional generalization capabilities, attaining a success rate of 44.3% in previously unseen positions.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy
cs.RO 2026-07 conditional novelty 6.0 of 10

RedFlow improves flow-matching VLA policies offline by labeling individual actions from failed rollouts with progress signals and pushing those actions toward successful alternatives found in similar contexts.
Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence
cs.AI 2026-04 unverdicted novelty 6.0 of 10

Projection-aware activation steering using logistic regression recovers honesty and compassion under malicious prompts while preserving coherence and benchmark performance better than uniform steering.
From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning
cs.RO 2026-03 accept novelty 6.0 of 10

Residual off-policy RL with selective BC regularization and value-guided sampling contracts a pretrained generative robot policy around successful actions, reaching high success on hard long-horizon tasks from pixels ...
ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Training
cs.RO 2026-02 conditional novelty 5.0 of 10

ALOE uses chunked TD bootstrapping with a pessimistic Q-ensemble to enable action-level off-policy value estimation for advantage-weighted post-training of flow-based VLA policies, reporting consistent success-rate ga...
Welcome New Doctor: Continual Learning with Expert Consultation and Autoregressive Inference for Whole Slide Image Analysis
cs.CV 2025-08 unverdicted novelty 5.0 of 10

COSFormer, a continual learning Transformer for whole slide image analysis, claims superior performance across seven datasets and six tasks without revisiting historical data.

Reference graph

Works this paper leans on

1 extracted references · 1 linked inside Pith · cited by 5 Pith papers

[1]

�� ...

arXiv 2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

�� ...

arXiv 2025