{"id":"98796b29-91f5-457d-9862-b6146b576d17","arxiv_id":"2601.00969","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding a value function trained on VLA rollout returns to VLAPS MCTS improves success rates in some LIBERO suites, but gains vary and are not consistently reported.","lead":"A team adds a small learned 'value head' to a vision-language-action robot planner, letting its tree search prefer branches predicted to end in success. In simulated manipulation tests, the value-guided planner improves success on some suites, but the headline gains are inconsistent between the abstract and the paper body.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The value head's training targets are generated by the same Octo policy it is meant to correct; the one held-out case that would demonstrate genuine generalization (spatial task 9) is absent from training data, and no ablation isolates the value signal's contribution.","rationale":"The reader's weakest assumption identifies the same core issue: value targets are generated by the policy being corrected, so the value head may just encode the policy's own preferences. I agree with that. The paper's strongest empirical counterexample is spatial task 9, which is absent from training and where the value head predicts much higher success than VLAPS. But if the value head generalizes across tasks, that would actually be good evidence; the problem is that the paper provides no direct evidence that the value predictions, rather than search noise or the U-term, cause the improvement on that task. A constant-value ablation and prediction-separation analysis would settle this. The abstract/body discrepancy (default vs. larger budget, LIBERO-10) further weakens confidence in the exact numbers but is secondary to the causal attribution issue; it is a reporting problem that could be fixed, whereas the attribution problem is fundamental to the method's premise. The reader's CONDITIONAL verdict is appropriate; no change needed, but the revision must add the ablation and clarify budgets.","tokens_in":8132,"tokens_out":9986,"duration_ms":217191,"concrete_test":"On spatial task 9 (held out from value-head training), run V-VLAPS with the trained value head vs. an identical pipeline with V_theta fixed to 0, using the same seeds, action library, and MCTS budget as Table 2; use at least 10 independent seeds and report success rates with confidence intervals. If the success rates are not significantly different (e.g., Fisher's exact test p>0.05), the reported 31-point gain is not attributable to the learned value signal. Also compute the separation (e.g., AUC) of predicted values for successful vs. failed MCTS rollouts on task 9; AUC near 0.5 would confirm the value head is not providing the signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that V_theta provides information beyond the VLAPS prior, not just that the overall V-VLAPS pipeline scores higher. Section 3.1 defines G_t = gamma^{T-t} (success) or 0 (failure) using rollouts of the fixed Octo policy with no planning. Therefore the MLP is trained to predict the same policy's own outcome/timing profile, and for states only reachable by actions the policy would not take, there are no targets from independent task progress. This makes the headline improvement untested as a value signal. The most important test case is spatial task 9: Octo has 0% success, task 9 is not in the training set (Section 4.1), yet Table 2 shows VLAPS 16% vs V-VLAPS(spatial) 47%. That 31-point jump is either a remarkable cross-task generalization of a 3-layer MLP over Octo readouts or an artifact of the U-term/exploration/noise. The paper provides no predicted-value distribution, no AUC, and no ablation with V_theta replaced by a constant for this task. Additionally, the abstract's 'larger search budget' and LIBERO-10 results do not appear in the full text, so it is unclear which configuration generated the headline numbers. Without isolating the value head's causal contribution on held-out tasks, the central claim is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes V-VLAPS, an extension of the VLAPS MCTS planner in which a lightweight MLP value head is trained on the frozen Octo VLA's latent readouts to predict Monte Carlo returns. The return targets are discounted sparse success rewards collected from rollouts of the fixed Octo policy. The predicted value is inserted into the VLAPS node-selection score as a bias term. Experiments on two LIBERO suites (Spatial and Object) report success-rate gains over the value-free VLAPS baseline (+5.2 and +2.8 points overall) and reductions in average MCTS simulations (5% and 14%), including a 31-point gain on a held-out spatial task. The abstract additionally claims default-budget matching and larger-budget gains including LIBERO-10, but those results do not appear in the body.","tokens_in":8541,"tokens_out":6989,"duration_ms":71017,"significance":"The core idea is timely and potentially useful: if a small value head on top of a frozen VLA backbone can improve and accelerate search, it would make VLA-guided planning more practical. The paper merits attention for framing the problem and for reporting a positive result on a public benchmark. The held-out task result (Spatial task 9) is the most interesting piece of evidence. However, the current manuscript is preliminary: the value signal is derived from the same policy being corrected, the main tables lack uncertainty quantification, the search score is under-specified, and the abstract reports experiments not present in the body. The manuscript's own limitation and future-work sections acknowledge the data-quality and training-signal issues, which supports the need for additional analysis rather than invalidating the approach.","major_comments":[{"comment":"The value targets are generated by rolling out the fixed Octo policy: G_t = gamma^{T-t} on success and 0 on failure. The value head is therefore trained to predict the same policy's own outcome/timing profile. This makes it possible that V-VLAPS improves by selecting states where Octo is already confident, rather than by correcting a biased prior. The paper's own future-work sentence ('we plan to guide data collection using VLAPS ... rather than VLA rollouts') concedes the limitation. To support the causal claim that V_theta provides information beyond the VLAPS prior, the paper should include an ablation replacing V_theta with a constant, a calibration/AUC analysis of predicted values, and a test on states reached by non-VLA actions. Without this, the headline gain is not attributable to the value signal.","section":"§3.1, §4.2, Table 2"},{"comment":"No error bars, confidence intervals, or significance tests are reported. The evaluation uses 10 rollouts per initial state (0–9), but the number of episodes per task is not stated. For Spatial task 9, the 31-point difference between VLAPS (16%) and V-VLAPS (47%) is presented as the key evidence of generalization; on a single held-out task this could be sampling noise. Please report per-task episode counts, standard errors, and a statistical comparison (e.g., bootstrap CIs or a paired test over initializations).","section":"§4.2, Table 2"},{"comment":"The abstract states that V-VLAPS 'matches value-free planning baseline at the default search budget in aggregate' and that with a larger budget it improves '+6 percentage points on LIBERO-Object and +4 percentage points on LIBERO-10'. The body reports no default-vs-larger-budget comparison, no LIBERO-10 results, and no aggregate matching; Table 2 shows gains at the (unspecified) budget used. This inconsistency makes it impossible to know which configuration produced the headline numbers. The abstract should be rewritten to match the experiments actually reported, or the missing experiments should be added.","section":"Abstract vs. §4.2"},{"comment":"The selection score is not fully specified. The text defines Q(v,a_i)=V_theta(h), then states SCORE = \\hat v_theta(readout(s')) + psi sqrt(N)/(1+N), without explaining whether the Q term is the predicted value of the next state only, whether a rollout/backup update is used, or how the value is combined with visit counts in already-expanded nodes. As written, the method is not reproducible and it is unclear whether the improvement comes from replacing Q with a learned value or from adding an extra exploration bonus. Please provide the complete score with all terms and the backup rule.","section":"§3.3"},{"comment":"The simulation-count claim is only partially supported. Table 3 shows a 5% reduction for Spatial with the spatial-only head and a 14% reduction for Object with the object-only head, but the jointly trained head increases simulations on Object (12.61 vs 11.95). The stated '5–15%' reduction therefore depends on which value head is selected. Also, no search budget is defined, so it is unclear whether the number of MCTS simulations is the controlled variable or an outcome; if it is an outcome, the success-rate comparison may confound compute with search depth.","section":"§4.1, Table 3"}],"minor_comments":[{"comment":"The formatting of several entries is broken, e.g., '7010098100' should be split into separate numbers (70 100 98 100). The column labels 'V-VLAPS (spatial)' and 'V-VLAPS (object)' are also ambiguous when the same table appears for both suites.","section":"Table 2"},{"comment":"The column header 'Avg Success Episode Length (VLA) in Steps' is confusing; specify whether this is the average episode length over successful episodes only, or over all episodes weighted by success.","section":"Table 1"},{"comment":"The t-SNE figure would be more informative if the color legend and the exact value-target binning were described. A quantitative separability measure (e.g., AUROC of a linear probe on readouts) would strengthen the claim that latent readouts encode success information.","section":"Figure 2"},{"comment":"The limitations paragraph is candid, but some of its content (e.g., 'we could only test on tasks from two of the LIBERO task suites') should be reflected in the abstract and introduction, which currently promise a broader evaluation.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"This is an early-stage result with a promising idea, but the current evidence is not up to journal standards. The main risks are the training-signal circularity and the abstract/body mismatch. I recommend a major revision focused on adding the missing ablation and uncertainty analysis; if the authors cannot supply the missing experiments, a reject may be appropriate. The paper is within scope for a robotics journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does one simple thing: train a small MLP on Octo latent readouts to predict success-return, then inject that prediction into VLAPS's PUCT score. If the effect is real, it's a useful trick—less reliance on the imitation prior, fewer MCTS simulations on average. The held-out spatial task 9 result (VLAPS 16% vs V-VLAPS 47%) is the single most interesting number in the paper.\n\nCredit where it's due: the method is clearly described, easy to reproduce from the text, and the limitations section is refreshingly honest about narrow scope, poor-quality rollouts, and the simulator assumption. The t-SNE and value-curve figures give qualitative support for the idea that VLA latents carry value-relevant structure. The contribution is incremental—AlphaGo-style value-guided MCTS applied to VLAPS—but it is a reasonable next step.\n\nThe problems are real and mostly about reporting. The abstract and full text tell different stories. The abstract says V-VLAPS \"matches baseline at the default search budget\" and only improves with a larger budget, citing +6 on Object and +4 on LIBERO-10. The full text reports single-budget numbers on Spatial and Object only, with no LIBERO-10 anywhere. That is a serious inconsistency that makes it unclear which configuration produced the headline results. There are no error bars, confidence intervals, or significance tests anywhere. The claim of \"over 5 percentage points\" improvement is true for Spatial (5.2) but not Object (2.8). The stress-test concern about the value head being trained on rollouts of the same policy it is meant to correct is not fatal—AlphaGo did something similar—but the paper does not show predicted-value distributions or an ablation replacing V_theta with a constant, so the causal contribution of the value signal on task 9 is untested. Minor point: Table 3 shows simulations increasing on several tasks, so the \"5-15% reduction\" is aggregate and should be labeled as such.\n\nWho is this for? Researchers working on VLA test-time planning or MCTS for manipulation. It is a methods note, not a foundational result. I would send it to peer review rather than desk-reject—the idea is sensible and the evaluation direction is right—but the authors need to reconcile the abstract with the body, add basic statistics, and ideally release code. If I were editing, I'd ask for major revision before acceptance.","headline":"A sensible, incremental idea—add an AlphaGo-style value head to VLAPS—reported with inconsistent numbers and too little statistical evidence for the headline claims.","tokens_in":8972,"tokens_out":2871,"would_cite":false,"duration_ms":28654,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A value head trained on a VLA's own rollout returns can guide tree search to raise manipulation success by over five points while using fewer simulations.","keywords":["vision-language-action","Monte Carlo tree search","value function","robot manipulation","test-time search","distribution shift","latent representations","imitation policy bias"],"falsifier":"A reader could settle the claim by collecting a held-out set of VLA states with known success/failure outcomes, then checking whether the learned value head ranks the successful states above the failed states on a task it never trained on; if the ranking is no better than random, the value signal is not the mechanism behind the reported gains.","tokens_in":8032,"feed_emoji":"🤖","tokens_out":7319,"duration_ms":66892,"temperature":0.7,"pith_summary":"The paper sets out to show that a pretrained vision-language-action (VLA) policy, which is normally used reactively, can be made more reliable by giving its Monte Carlo tree search a learned notion of which states lead to success. The authors claim that a lightweight three-layer MLP, trained only on the policy's own rollout returns and keyed to the policy's latent readouts, provides that signal. Folding this value into the search score improves success rates by 5.2 points on a spatial manipulation suite and 2.8 points on an object suite, while reducing the average number of search simulations by roughly 5 to 14 percent. The improvement holds even for a hard task that was absent from the value head's training data, which the authors read as evidence that VLA latent states carry reusable value information.","feed_headline":"Value-guided tree search lifts robot success by 5+ points","feed_subtitle":"Training an MLP on a VLA's own rollouts lets Monte Carlo search correct imitation bias and cut simulations up to 14%.","key_machinery":"The value head is a three-layer MLP that maps the VLA's last-layer latent readout to a scalar estimate of the discounted Monte Carlo return. Its predictions enter the tree-search selection score as the Q term, so a branch is favoured when the predicted value of the state it leads to is high, in addition to the usual exploration bonus. The training targets are discounted returns from rollouts of the frozen policy, with a sparse terminal reward of one for success.","core_discovery":"The central claim is that the latent representations of a frozen vision-language-action policy already carry enough information about future task success that a small regression head can turn them into a usable value estimate for planning. The paper demonstrates this by training a three-layer MLP on Monte Carlo return targets derived from rollouts of the same policy, then inserting the predicted value into the PUCT selection rule as the Q-term. In the resulting search, node selection is driven by predicted value plus the VLAPS exploration bonus rather than by visit counts and the policy prior alone. On two manipulation task suites the value-guided variant outperforms the value-free baseline,","pith_inferences":["The mechanism is representation-agnostic: because the value head consumes only the latent readout, the same recipe could be applied to any VLA model that exposes a fixed-dimensional state summary, not just the one used in the paper.","If the value head truly generalizes across tasks within a suite, then value models could be trained once per task family and reused for new tasks, sidestepping the need for per-task reward engineering.","The authors' own proposal to train the value head on search-generated rollouts rather than policy rollouts is a likely next step; if search data yields sharper value estimates, the improvement over the baseline might grow beyond the 2-5 point range reported here.","Because the paper only tests two suites, the claim that VLA latents are 'value-ready' is still narrow; a deliberate test would be to see whether the same value-head setup transfers to more diverse tasks such as long-horizon assembly or tasks with partial observability."],"forward_implications":["If the central claim holds, any frozen VLA model can be retrofitted with a value head trained from its own rollouts, turning test-time search into a way to correct, not just exploit, the imitation prior.","With a larger search budget, the value-guided planner improves over the value-free baseline in all tested suites, including a six-point gain on the object suite and a four-point gain on a ten-task suite.","The value head transfers to tasks it never saw during training, as shown by a 31-point improvement on a hard spatial task absent from the training data.","Adding the value head reduces the average number of MCTS simulations by about 5 percent on the spatial suite and 14 percent on the object suite, meaning the value signal concentrates search on useful branches.","Training a single value head on both task suites did not outperform per-suite heads, suggesting value generalization across task families is an open problem."],"fun_headline_variants":["Value-guided MCTS beats VLA baseline at larger search budgets","Train a value head on VLA rollouts to guide MCTS and win more","VLA representations encode returns: small head improves planning","V-VLAPS: value-guided search lifts LIBERO success with extra compute"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The value head is trained on rollouts of the same policy it is meant to correct, so it may merely be encoding the policy's existing preferences rather than providing an independent measure of task progress.","fun_headline_variants_meta":{"raw":{"variants":["Value-guided MCTS beats VLA baseline at larger search budgets","Train a value head on VLA rollouts to guide MCTS and win more","VLA representations encode returns: small head improves planning","V-VLAPS: value-guided search lifts LIBERO success with extra compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000429,"raw_usage":{"total_tokens":2045,"prompt_tokens":772,"completion_tokens":1273,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1197}},"tokens_in":516,"tokens_out":1273,"duration_ms":11795,"temperature":1.0,"reasoning_tokens":1197,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T12:54:24.575800+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the claim by collecting a held-out set of VLA states with known success/failure outcomes, then checking whether the learned value head ranks the successful states above the failed states on a task it never trained on; if the ranking is no better than random, the value signal is not the mechanism behind the reported gains.","supporting_citations":[],"review_version":2}