{"id":"631817ef-70c2-4a50-ae87-c128c7100177","arxiv_id":"2505.03181","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AFSFT, an off-policy RL update that masks tokens with low estimated advantage, lets open-weight VLMs learn browser and game tasks from noisy or random data, improving over SFT.","lead":"Vision-language models can learn computer and game tasks from low-quality data using a new fine-tuning method that filters out bad actions with a learned value model. The method, called AFSFT, may make VLM agents cheaper and more reliable than standard supervised fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 5-6 require constraining Q to dataset actions, but Section 5.2 only states the constraint is necessary and no mechanism is specified; this unverified step is load-bearing.","rationale":"I read the paper as claiming a single-VLM offline-to-online actor-critic update in which a learned Q-head filters token-level actions so AFSFT improves over SFT on suboptimal data. The strongest support is the consistent MiniWoB/Gym Cards/BabyAI results and the clear equations. The weakest point is exactly where the method departs from standard CRR: the advantage baseline and TD target are expectations over a 51k-257k token vocabulary, while Q-values for most of that vocabulary are never trained. The paper itself flags this in Section 5.2 and says a dataset-support constraint is necessary, but no such mask appears in Eq. 5 or Eq. 6 and none is described in Appendix A.2. If the literal equations are executed, the filter is contaminated by random Q-values; if they are not, the paper omits a core mechanism. This is not a disagreement with consensus; it is an unverified implementation detail at the center of the contribution. The proposed ablation would settle it. I therefore leave the reader's CONDITIONAL verdict unchanged.","tokens_in":22781,"tokens_out":4695,"duration_ms":51535,"concrete_test":"Re-run the MiniWoB MoonDream2+AFSFT pipeline on the 340k-action offline dataset with instrumented logging of the support of Qψ(h)^T Lφ(h) in Eq. 5 and of y in Eq. 6, then compare two ablations: (A) full-vocabulary expectation exactly as written; (B) expectation masked to tokens that appear in the offline dataset (equivalently, valid BrowserGym action tokens). Report per-task success rate, valid-action rate, and fraction of filtered tokens across the 27 tasks. If A and B differ materially, the unconstrained random-Q baseline is load-bearing and the paper's stated 'necessary' constraint was not enforced; if they match, the constraint is effectively present and should be documented as the actual algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (AFSFT reliably replaces SFT on suboptimal data) requires the critic in Eq. 5 to rank tokens by true task value. Section 5.2 explicitly concedes that Qψ outputs for tokens absent from the dataset are random and states that constraining updates to dataset actions 'is necessary,' even online. However, neither the filter threshold Qψ(h)^T Lφ(h) nor the TD target y in Eq. 6 is written with such a mask, and no implementation detail describes one. Literally, the baseline is an expectation over the full |V|-token vocabulary; any probability mass Lφ puts on unseen tokens contributes random state-dependent noise to the advantage threshold and to every bootstrap target, so the filter could mask tokens for reasons unrelated to value. If the implementation does restrict the expectation to seen or valid tokens, that mechanism is absent from the paper, making the central experimental result unverifiable at the exact point where the method differs from SFT. The MiniWoB results do show AFSFT beating SFT, and the paper is honest about the issue, but the missing constraint is the load-bearing uncertainty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VLM Q-Learning, an offline-to-online reinforcement learning method for fine-tuning vision-language models as interactive agents. The core contribution is Advantage Filtered SFT (AFSFT), which adds a critic output head to a VLM, converts each turn of dialogue into token-level RL transitions, and masks low-advantage tokens during supervised fine-tuning so that the policy can improve beyond noisy or suboptimal demonstrations. The method is evaluated with MoonDream2 and xGen-MM on Gym Cards, BabyAI, and 27 MiniWoB click tasks, with additional LLM baselines and comparisons against SFT from the same noisy dataset. The paper argues that AFSFT is a low-risk replacement for SFT in VLM agent training and enables a smooth transition to online learning without a separate PPO stage.","tokens_in":22992,"tokens_out":4481,"duration_ms":44968,"significance":"If the method performs as claimed, it offers a practical, single-model alternative to SFT for VLM agent training, with the notable property of learning from mixed-quality data and improving beyond the demonstrations. The paper's strengths include a clear reduction of turn-based agent interaction to token-level RL, the use of open-weight models across three distinct domains, and an honest discussion of the large-vocabulary Q-value calibration problem. The MiniWoB results, in particular, show AFSFT clearly outperforming SFT trained on the same noisy data, which is a meaningful empirical contribution. However, the central mechanism's specification is incomplete at the exact point where AFSFT differs from SFT, and the experimental evaluation lacks variance estimates, so the strength of the central claim is currently not fully verifiable.","major_comments":[{"comment":"The advantage filter and the TD target are written over the full token vocabulary: f(s,a)=Q(s,a)-Σ_k Q(s,k)L(k)-β in Eq. (5) and y=r+γ(1-d)∇(Q(h')^T L(h')) in Eq. (6). The paper acknowledges that Q-values for tokens never seen in the dataset are random and states that 'this constraint is necessary even when we do allow for online interaction,' but no masking or constraint is shown in the equations, the training loop, or the hyperparameter tables. As written, the baseline expectation and the bootstrap target include probability mass on unseen tokens, injecting random state-dependent noise into every advantage decision and every target. The authors must specify exactly whether and how the expectation and the TD target are masked to the set of actions present in the dataset (or to a valid-action mask), and if no mask is used, they must explain why the stated necessity does not invalidate Eqs. (5)-(6). This is load-bearing because the filter and the critic are the core of AFSFT.","section":"§5.2, Eqs. (5)-(6)"},{"comment":"The experimental results are presented as point estimates from what appear to be single training runs and single evaluation samples. For MiniWoB, Table 5 states that success rates are computed over a sample of 5,000 timesteps per task; many tasks have 0.0% success for several methods, and the headline comparison (MD2+AFSFT vs MD2+SFT) is a contrast of two point estimates without variance, confidence intervals, or a paired test across the 27 tasks. The conclusion that AFSFT 'can effectively replace SFT' in suboptimal-data scenarios is a strong empirical claim and should be supported by multiple seeds (at least 3-5) with error bars or interquartile ranges, or by a significance test over tasks. This is essential to rule out that the reported gains over SFT are due to a single lucky run or evaluation sample.","section":"Figures 5-7 and Tables 5-6"},{"comment":"The TD target in Eq. (6) treats h' = stopgrad-VLM(o_{t+1}) as a fixed state for bootstrapping future value. However, h' is the VLM's own representation of the next observation and it changes as the VLM is fine-tuned with L_VLMQ; it is not a stationary environment state. The paper does not explain why the one-step bootstrap remains calibrated under this non-stationarity, nor does it provide diagnostics such as TD error, comparison with Monte Carlo returns, or sensitivity to the stop-gradient choice. Since the filter's correctness relies on calibrated Q-values, this is a correctness risk that should be addressed with an experiment or formal argument, at least for the environments studied.","section":"§5.2, Eqs. (2) and (6)"}],"minor_comments":[{"comment":"The notation in Eq. (2), '∇VLMθ(o_{t+1})', appears to be a typographical artifact; the stop-gradient operator is not clearly defined. Please clarify the notation so that the reader can tell which parameters are updated through h'.","section":"§5.2, Eq. (2)"},{"comment":"The advantage threshold β is said to be initialized to -∞ and then scheduled to reach 0 'after a few hundred gradient steps.' Please specify the exact schedule (number of steps, interpolation method) and whether it is the same across all domains and runs.","section":"§5.2, paragraph on offline-to-online fine-tuning"},{"comment":"PopArt normalization is listed as an implementation detail, but Eq. (6) does not show where the normalization is applied. Please indicate whether the TD loss is computed on normalized values and whether the filter in Eq. (5) uses the raw or normalized Q-output.","section":"Appendix A.2.1, Table 3"},{"comment":"The comparison against 'a reference score for a similar setup in Zhai et al. (2024)' should state whether this is the published RL4VLM number or a recomputed baseline, and how the observation prompts differ across the two setups.","section":"Figure 5 caption and Section 6"}],"recommendation":"major_revision","confidential_remarks":"The missing masking detail in Eqs. (5)-(6) is the key uncertainty for me. The paper's own language suggests the authors are aware of the problem, but without the actual mechanism the central claim cannot be reproduced or verified from the text. If the implementation does restrict the baseline and target to seen actions, this is a simple but essential addition to the method description. I would also encourage the authors to add seed variance to the MiniWoB experiments; given that many tasks are at 0% success, the current point estimates are not persuasive on their own. The workshop version may have had space constraints, but for a journal venue these details are necessary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. The core idea is genuinely useful: adding a critic head to a VLM and using CRR-style advantage filtering as a drop-in replacement for SFT is a practical trick, and it works in the experiments. MiniWoB is the cleanest demonstration: AFSFT takes MoonDream2 from 0 to roughly 66% success on noisy data where SFT only reaches about 17%. The second thing is that the paper is honest about its weakest point but does not close it. Section 5.2 admits that Q-values for tokens absent from the dataset are random and says constraining updates to dataset actions is necessary, even online. Neither the equations nor the appendix specify how that constraint is enforced. That is load-bearing: the filter in Eq. 5 and the TD target in Eq. 6 both average Q-values over the full vocabulary via the policy. If the implementation masks or restricts that expectation to seen tokens, the mechanism is missing from the write-up; if it does not, the filter is partially driven by random numbers. The authors need to say which.\n\nWhat is new is not the principle: CRR, ILQL, and ArCHer already cover the pieces. The contribution is the adaptation — a single VLM with a shared backbone, one language head and one critic head, trained jointly with an SFT loss plus a one-step TD loss, and the critic used as a hard advantage filter over token actions. The positioning against RL4VLM is fair, and the appendix has enough hyperparameters to reimplement.\n\nSoft spots, in order. The constraint gap above is the most important. Beyond that, there are no multiple seeds or error bars anywhere; the MiniWoB success rates come from a single 5,000-timestep sample per task; and Gym Cards and BabyAI have no direct SFT baseline trained on the same data, so the only clean SFT comparison is MiniWoB. No code or data is released, which matters when the key implementation detail is absent. All of these are addressable and none contradict the central claim.\n\nThis is for people building open-weight VLM agents who want a cheap offline-to-online RL alternative to SFT. It is not a new RL principle, but it is clearly thought through. I would send it to peer review as a conditional: require the masking mechanism to be specified, add seeds and error bars, and run SFT on all three domains. The idea is worth refereeing, not desk rejecting.","headline":"AFSFT is a practical single-model RL alternative to SFT for VLM agents, but the missing unseen-token constraint and thin experimental reporting make it a conditional accept.","tokens_in":23560,"tokens_out":3214,"would_cite":true,"duration_ms":29250,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single VLM with an added critic head can learn agent policies from mixed-quality offline data and keep improving online.","keywords":["vision-language models","offline reinforcement learning","advantage-filtered supervised fine-tuning","Q-learning","token-level action spaces","VLM agents","offline-to-online RL","supervised fine-tuning"],"falsifier":"Train AFSFT on a Gym Cards dataset of purely random valid actions with $\\beta = 0$, and check whether the final policy's success rate exceeds the best trajectory in the dataset; the paper's claim predicts it should. Separately, randomly permute $Q_\\psi$ outputs for tokens that never appear in the training buffer before computing Eq. 5: if the filtered policy and success rate are unchanged, the method is effectively constrained to seen actions, while a measurable change would reveal an unconstrained filter operating on uncalibrated values.","tokens_in":22535,"feed_emoji":"🧠","tokens_out":9391,"duration_ms":90835,"temperature":0.7,"pith_summary":"This paper claims that supervised fine-tuning (SFT) for vision-language agents can be replaced by advantage-filtered supervised fine-tuning (AFSFT), an offline reinforcement learning update that masks tokens predicted to reduce downstream performance. The method adds one small critic head to a single VLM, treats each output token as an action, and trains the language head only on tokens whose estimated advantage clears a threshold. The claim is that this preserves the stability and simplicity of SFT while letting the agent outperform its demonstrations, learn from noisy or suboptimal data, and transition to online learning without a separate SFT-then-PPO pipeline. If true, any team already doing SFT for VLM agents could switch to AFSFT and gain self-improvement at almost no architectural cost.","feed_headline":"One extra output head lets VLM agents outlearn their demonstrations","feed_subtitle":"Advantage-filtered supervised fine-tuning replaces SFT and skips the separate PPO stage.","key_machinery":"The load-bearing mechanism is the turn-to-token conversion paired with a second critic output head. The VLM's hidden representation $h_j$ is the RL state, the sampled token is the action, and the next state is either the next token's hidden state or a stop-gradient representation of the next turn; the critic's one-step temporal-difference bootstrap (Eq. 6) then supplies the Q-values that the advantage filter (Eq. 5) uses to mask the behavior-cloning loss (Eq. 4). This setup makes AFSFT literally SFT plus a learned mask, so it inherits SFT's stability while adding a way to reject suboptimal decisions.","core_discovery":"The central discovery is that VLM agent fine-tuning can be reframed as an off-policy actor-critic problem in which each token of the reply is a discrete action. A turn is expanded into token-level transitions using the VLM's hidden representations as states, a second output head $Q_\\psi$ estimates the value of every token, and the advantage filter $f(h,a) = Q_\\psi(h)[a] - Q_\\psi(h)^\\top L_\\phi(h) > \\beta$ decides which tokens the language head should imitate. The actor optimizes a masked next-token loss and the critic optimizes a one-step TD loss, both within one VLM. In experiments, this lets two open-weight VLMs match or exceed the RL4VLM reference scores in Gym Cards, reach near-perfect success in an online BabyAI task, and recover a competitive MoonDream2 browser agent from a noisy 340k-action MiniWoB dataset built partly from invalid action prefixes.","pith_inferences":["The success on partial-action MiniWoB datasets suggests the advantage filter may be learning action syntax itself, not just semantic action quality; a useful ablation would separate the gain from filtering syntactically invalid tokens from the gain from filtering valid-but-suboptimal ones.","Because the paper leaves the constraint on the advantage filter unspecified, an explicit masking of both Q-values and the policy-weighted baseline to the dataset's token support would make the method more principled and likely more reproducible.","If the token-level TD bootstrap genuinely works with next-turn hidden representations as states, the same recipe could extend to any sequence-output policy, including open-ended text actions, without needing a hand-designed environment parser.","The paper's own claim that noisy critics reduce AFSFT to random-subset SFT implies that the method's advantage over SFT is bounded by critic calibration, so measuring calibration quality directly would predict when AFSFT helps and when it is neutral."],"forward_implications":["AFSFT can replace SFT in VLM agent pipelines whenever the demonstration dataset is suboptimal or the action syntax is unreliable, removing the need for a separate syntax-alignment SFT stage.","A single VLM can act as both policy and value model, so practitioners avoid the cost and complexity of training separate actor and critic networks or a reference policy.","The same loss works offline and online, so an agent can start from a static buffer and continue improving with its own collected experience without switching algorithms.","When the critic is noisy or near initialization, the update degrades gracefully toward SFT on random subsets of the data, giving the method a low downside relative to plain SFT.","Token vocabularies are large enough that Q-values for unseen tokens are unreliable, so constraining critic updates to actions observed in the buffer is necessary even with online interaction."],"supporting_citations":[{"why":"Supplies the Critic Regularized Regression offline RL method that the advantage filter in AFSFT is most similar to.","marker":"Wang et al. (2020)"},{"why":"Provides the RL4VLM two-stage SFT-then-PPO baseline and the Gym Cards benchmark whose reference scores the experiments match or exceed.","marker":"Zhai et al. (2024)"},{"why":"Establishes token-level offline RL for language generation and motivates the large-vocabulary Q-value miscalibration problem.","marker":"Snell et al. (2022)"},{"why":"Introduces the turn-based versus token-based action distinction that the paper's state construction builds on.","marker":"Zhou et al. (2024)"},{"why":"Frames offline RL and explains how filtered behavior cloning can outperform the demonstrations in a static dataset.","marker":"Levine et al. (2020)"},{"why":"Provides the moving-average target critic used to stabilize the one-step TD bootstrap in Eq. 6.","marker":"Lillicrap (2015)"},{"why":"Supplies PopArt normalization that scales the critic's Q-value predictions during training.","marker":"van Hasselt et al. (2016)"},{"why":"Supplies the LoRA adapters that let the agent discard task-specific weights and recover the base VLM, making the single-model approach practical.","marker":"Hu et al. (2022)"}],"fun_headline_variants":["Off-policy RL with advantage filter beats SFT for VLM agents","Token-level Q-head lets VLMs learn from mixed-quality data","VLM agents self-improve via advantage-filtered off-policy RL","One extra head converts SFT into off-policy RL for VLM agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the critic's predicted future returns (Q-values) are calibrated well enough to choose which tokens to imitate, even though the critic outputs a value for every token in the vocabulary and the paper itself notes that values for tokens never seen in the dataset are random; if the advantage filter is not actually restricted to actions that appeared in the data, token selection could be driven by noise rather than by task value.","fun_headline_variants_meta":{"raw":{"variants":["Off-policy RL with advantage filter beats SFT for VLM agents","Token-level Q-head lets VLMs learn from mixed-quality data","VLM agents self-improve via advantage-filtered off-policy RL","One extra head converts SFT into off-policy RL for VLM agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000952,"raw_usage":{"total_tokens":4057,"prompt_tokens":941,"completion_tokens":3116,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":3050}},"tokens_in":557,"tokens_out":3116,"duration_ms":23218,"temperature":1.0,"reasoning_tokens":3050,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:57:54.952104+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train AFSFT on a Gym Cards dataset of purely random valid actions with $\\beta = 0$, and check whether the final policy's success rate exceeds the best trajectory in the dataset; the paper's claim predicts it should. Separately, randomly permute $Q_\\psi$ outputs for tokens that never appear in the training buffer before computing Eq. 5: if the filtered policy and success rate are unchanged, the method is effectively constrained to seen actions, while a measurable change would reveal an unconstrained filter operating on uncalibrated values.","supporting_citations":[],"review_version":1}