{"id":"b307d549-7c24-4956-9fd1-47a4ab37eb7c","arxiv_id":"2509.02722","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A vision-language world model trained on 180k instructional and egocentric videos predicts action-state trajectories in language, and a self-supervised critic selects plans by cost minimization.","lead":"This paper trains an 8B vision-language model to predict goals, actions, and world-state changes from video as text, then adds a critic that scores candidate plans and picks the lowest cost. It reports sizeable gains on video planning benchmarks and a +27% Elo improvement in a human preference arena.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified LLM-extracted state labels make the 'world model' claim non-identifiable: VLWM may be optimizing text-style fluency, not physical dynamics.","rationale":"I read the paper as a systems contribution, and much is done well: the Tree-of-Captions compression, the explicit Eq. 1 objective, the self-supervised critic, and the independent OGP/WorldPrediction evaluations are real evidence. The reason I do not call for rejection is that the pipeline is coherent and the external critic results provide partial support. But the world-model claim is not yet separable from text-style imitation. The reader's weakest assumption was extraction faithfulness; I agree, and sharpen it by pointing to the explicit contradiction in Appendix B.2 and the circularity of critic training/evaluation. A human-verification subsample is the decisive missing measurement. If it passes, the paper should be treated as CONDITIONAL with that caveat; if it fails, the central claim would need to be reframed as 'language-consistent planning' rather than 'world modeling'.","tokens_in":24889,"tokens_out":6042,"duration_ms":74824,"concrete_test":"Human-verify a stratified random sample of 200 extracted (action, ΔS) steps from the six training sources (e.g., 25 per source). For each step, play the corresponding video segment; ask three annotators (i) whether the action is actually performed in the segment, (ii) whether each concrete state claim (object, position, temperature, readiness, enablement) is visibly supported or entailed, and (iii) whether any state text names objects/events absent from the segment. Report per-step precision and Fleiss' κ. If per-step precision <90%, the labels are too unreliable to support the claim that VLWM learns true dynamics; if ≥90%, the central assumption survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. 1 trains VLWM to output interleaved action/state trajectories that are generated, not measured. §2.1.2 uses Llama-4 Self-Refine on Tree-of-Captions to produce 5.7M steps; the only faithfulness check is a prompt instruction in Appendix B.2. That instruction is self-contradictory: it says 'Do not hallucinate or infer based on commonsense knowledge', yet the same prompt requires each state paragraph to discuss 'motivation, rationale, and purpose', 'what the change enables', and 'whether it is ready for future steps'—exactly commonsense elaborations that are not observable in the video. Consequently the regression targets are plausible text about states, not grounded state observations. The critic (§2.2.1) is trained on the same extracted pipeline (valid next steps vs distractors/shuffles), so system-2 cost minimization may select Self-Refine-typical text rather than physically valid progress. The OGP and WorldPrediction-PP evaluations are genuinely independent for the critic, but they do not verify VLWM's predicted ΔS against real state changes. Thus the headline results establish language-consistent planning, but the central 'world model that understands physical dynamics' claim rests on an unverified label source.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VLWM, a vision-language model trained, given a config and visual context, to predict a goal description, a goal interpretation, and an interleaved action/world-state trajectory of the form specified in Eq. (1). Training targets are produced by compressing videos into a Tree of Captions, then using Llama-4 Self-Refine to extract structured plans. At inference, system-1 decodes plans autoregressively; system-2 generates multiple candidate rollouts and selects the plan with the lowest cost assigned by a critic trained with ranking losses on valid versus distractor/shuffled steps. The paper evaluates on VPA benchmarks (COIN and CrossTask), a new PlannerArena human-preference study, RoboVQA, critic goal-achievement detection, and WorldPrediction-PP, claiming state-of-the-art VPA performance, a +27% Elo improvement for system-2 over system-1, and top results on RoboVQA and WorldPrediction.","tokens_in":25234,"tokens_out":6853,"duration_ms":74028,"significance":"If the central claim is established, the contribution is significant: a scalable recipe for training high-level, language-based world models from uncurated video, with a self-supervised critic enabling test-time search. The pipeline has several strengths: the Tree-of-Captions compression is compute-efficient; the training corpus is large; several evaluations (OGP, WorldPrediction-PP, RoboVQA) are external to the training distribution; and the qualitative cost-minimizing versus cost-maximizing examples in Appendix D.2 are convincing demonstrations of the critic's behavior. However, the central construct-validity claim—that VLWM learns physical dynamics rather than text-style fluency—is not established by the current experiments, because the state labels are generated by the same LLM pipeline used for target construction and the benchmarks largely evaluate action ordering or language plausibility. The paper also overstates the VPA result and lacks statistical support for the PlannerArena Elo claim.","major_comments":[{"comment":"The training targets of Eq. (1) are generated by Llama-4 Self-Refine from PerceptionLM captions. The only faithfulness check is the prompt instruction in Appendix B.2, which is self-contradictory: it forbids hallucinating or inferring from commonsense knowledge, yet it requires each world-state paragraph to discuss 'motivation, rationale, and purpose', 'what the change enables', and whether it is 'ready for future steps'—content not observable in the video. Consequently, the predicted ΔS are not grounded state annotations, and the 'world model that understands physical dynamics' claim is non-identifiable: VLWM may be optimizing text-style fluency rather than learning genuine world dynamics. The authors should either provide external verification of predicted state changes (e.g., a human-annotated sample of state transitions, or a benchmark checking predicted ΔS against measured states) o","section":"§2.1.2, Appendix B.2, Eq. (1)"},{"comment":"The critic is trained on positive/negative pairs constructed from the same Self-Refine trajectories, and the in-domain goal-achievement evaluation (VLWM-Instruct and VLWM-Ego) uses trajectories and distractors drawn from that same distribution. The near-perfect accuracy (98.4% and 92.7%) therefore partly reflects self-consistency of the label-generation pipeline, not goal achievement. The OGP and WorldPrediction-PP evaluations are more independent, but OGP is action-only and WorldPrediction-PP evaluates the critic on externally provided captions, not on VLWM's predicted ΔS. The current experiments do not break the circularity for the system-2 planning claim. Please add an evaluation with human-verified or externally annotated state changes, or an ablation with an alternative label source.","section":"§2.2.1, §3.4.1, Table 5"},{"comment":"The abstract claims 'state-of-the-art VPA performance on both benchmark evaluations,' but Table 2 shows VLWM trailing VidAssist on 4 of 12 metrics: CrossTask T=3 mAcc 36.4 vs. 36.7; T=3 mIoU 48.3 vs. 48.9; T=4 SR 7.2 vs. 7.4; T=4 mIoU 51.1 vs. 51.6. The claim should be qualified (e.g., average improvement, 8 of 12 metrics). In addition, no error bars or significance tests are reported; several margins are small, so the superiority is not established with the current evidence. Please report variance across seeds/initializations and adjust the abstract accordingly.","section":"Table 2, Abstract"},{"comment":"PlannerArena is based on only 550 battle pairs with five annotators, and Elo scores are reported without confidence intervals. With K=32, the claimed +27% Elo improvement of system-2 over system-1 (1261 vs. 992) may be within sampling noise. Inter-annotator agreement is computed on only 100 samples (Fleiss' κ=0.63). The claim that system-2 improves over system-1 by +27% Elo needs bootstrap confidence intervals or a significance test before it can be considered load-bearing.","section":"§3.2.2, Table 3"}],"minor_comments":[{"comment":"'32 frames in 4482 resolution' is unclear; probably a typo for 448×448 or 448². Please clarify.","section":"§3.1.1"},{"comment":"Table header says 'Statics' and 'EPIC-KITHCNES-100'; both should be corrected to 'Statistics' and 'EPIC-KITCHENS-100'.","section":"Table 1"},{"comment":"'procedual' in the description of VPA benchmarks should be 'procedural'.","section":"§3.2.1"},{"comment":"Several references are incomplete or malformed: 'RoboMamba' lacks a paper title, and the OpenOrca entry has a URL beginning 'https://https://'. Please check all entries.","section":"References"},{"comment":"The text reports BLEU-4 values (VLWM 55.6 vs. RoboBrain 55.1), but Table 4 only lists BLEU-1. Either add the multi-gram columns or align the text with the table.","section":"§3.3, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The construct-validity concern about the LLM-generated state labels is the core issue. If the authors can add a human-verified sample of state-change predictions or reframe the contribution as a language-consistent planner, the paper could be a solid systems contribution. The abstract and 'state-of-the-art' phrasing also need tempering. I would not reject based on disagreement with the LLM-as-world-model paradigm, but the absence of any external check on ΔS is a gap that currently prevents the central claim from being supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Rough take: this is a genuinely integrated system — large-scale VLM trained on 1.2M LLM-extracted action-state trajectories, plus a self-supervised critic for cost-minimizing planning. That combination is new, and the paper is unusually complete on training recipe and evaluation coverage. It deserves serious referee time. But the abstract overstates the VPA result, and the 'world model' claim is only as good as the label source, which remains unverified.\n\nWhat the paper does well: the Tree-of-Captions compression with hierarchical feature clustering is sensible. The Self-Refine extraction at scale is a real engineering contribution. The critic trained on ranking valid versus distractor/shuffled continuations is a clean idea, and the OGP and WorldPrediction-PP evaluations are genuinely out-of-distribution for the critic. The PlannerArena human study, though small, shows a meaningful Elo gap between cost-minimizing and cost-maximizing, and system-2 beats system-1 — that internal contrast is a nice control.\n\nSoft spots: first, the abstract claims state-of-the-art on VPA, but Table 2 shows VLWM trails VidAssist on 4 of 12 metrics. The average gains are real (+3.2 SR, +3.9 mAcc, +2.9 mIoU), but the claim should be 'on most metrics' or 'on average'. Second, the label faithfulness issue. The Self-Refine prompt says 'Do not hallucinate or infer based on commonsense knowledge', but it simultaneously requires discussions of motivation, rationale, purpose, and what changes enable — exactly commonsense elaborations not observable in pixels. So the regression targets are plausible language about states, not measured states. The critic is trained on the same extraction distribution, so its high goal-achievement accuracy on in-domain data partly reflects self-consistency. The OGP and WorldPrediction-PP results do show the critic generalizes beyond that distribution, which is genuinely reassuring, but they do not verify that VLWM's predicted ΔS matches real physical state changes. So the strongest defensible claim is 'language-consistent planning', not 'physically grounded dynamics'. Third, there are no error bars or significance tests anywhere, and the 550-battle PlannerArena with five annotators is a modest basis for the Elo headline. Fourth, no code or data are available yet despite the 'will be open-sourced' promise.\n\nWho this is for: researchers working on vision-language planning, world models, and self-supervised critics. They will get a well-specified baseline and a useful dissection of the design space. I would send it to review, but the authors need to tighten the claims, either verify state predictions on a small labeled set or explicitly qualify the world-model claim, and release artifacts.","headline":"Solid systems paper on language-based world modeling, but the central 'understands physical dynamics' claim rests on LLM-generated state labels that are never verified against real state changes.","tokens_in":25691,"tokens_out":3414,"would_cite":true,"duration_ms":35001,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a world model trained on unlabeled videos to predict goals and action-state trajectories in natural language beats much larger prompting-based planners, and that a self-supervised cost-minimizing critic makes it better stil","keywords":["vision language world model","world modeling","procedural planning","language-based abstraction","cost minimization","self-supervised critic","Tree of Captions","visual planning for assistance"],"falsifier":"Take held-out instructional videos, adversarially replace the state-description portion of extracted training targets with descriptions from a different task while keeping actions and visuals intact, then test whether VLWM rollouts and critic cost rankings degrade; if they do not, the state descriptions are not doing the claimed dynamics work. A cheaper check: have humans compare VLWM's predicted world-state text against the actual next visual frames.","tokens_in":24809,"feed_emoji":"🧠","tokens_out":5064,"duration_ms":49693,"temperature":0.7,"pith_summary":"This paper tries to show that a high-level world model for planning can be learned directly from unlabeled videos, with the future represented in natural language instead of pixels or latent vectors. The Vision Language World Model (VLWM) is trained to turn a video context into a goal description plus an interleaved sequence of actions and world-state changes, using an automatic pipeline that compresses video into a Tree of Captions and refines it with LLM self-refinement. On top of this predictive model, the paper adds a self-supervised critic that scores how close a predicted action-state rollout comes to the goal, turning plan generation into cost minimization (system-2 planning). The paper reports state-of-the-art Visual Planning for Assistance results, a +27% Elo gain in human plan-preference evaluation, and strong results on RoboVQA and WorldPrediction, arguing that language-based world states are an efficient and interpretable substrate for long-horizon planning.","feed_headline":"A word-based world model beats 70B planners at 8B","feed_subtitle":"VLWM predicts future states in language, then picks plans by lowest cost; human raters prefer it 2-to-1.","key_machinery":"Tree of Captions: a hierarchy of video captions over adaptively segmented windows, produced by hierarchical agglomerative clustering of Perception Encoder features and PerceptionLM captioning; it compresses 1.1 TB of Ego4D video into under 900 MB of text. LLM Self-Refine extracts the goal/interpretation/action-state training targets from this tree. The critic is a 1B language model trained with a ranking loss (Eq. 2) to assign lower cost to valid goal progress and higher cost to distractors or shuffled steps; it supplies the semantic-distance cost for system-2 plan search.","core_discovery":"The central claim is Eq. 1: conditioned on visual context and a goal config, VLWM predicts the goal's interpretation and a trajectory of interleaved actions and world-state deltas <A0, ΔS0>, ..., <AN, ΔSN>. This single objective bundles three levels of world modeling — goal inference, action anticipation, and action-conditioned dynamics prediction — so the same model can act reactively (system-1) by decoding a plan, or reflectively (system-2) by rolling out multiple candidate actions and selecting the one with lowest cost assigned by a separately trained critic. The paper argues that generating language descriptions of state changes, rather than raw future frames, sidesteps pixel-level ill-p","pith_inferences":["A test the authors did not run: whether the world-state descriptions are genuinely grounded predictions or fluent text that leaks the plan; a caption-vs-state correlation check on held-out videos would settle it.","The critic's learned cost could plausibly double as a dense reward for RL fine-tuning of the planner — the paper uses it only for selection, not for further training.","Language as the state bottleneck means planning quality is capped by the descriptive coverage of the Tree of Captions and the Self-Refine extraction; a weaker captioner should degrade planning in a predictable, measurable way.","System-2 search width (20 candidates here) is a compute-accuracy dial; plotting Elo versus candidate count would show where additional rollouts stop paying off."],"forward_implications":["If VLWM is correct, high-level planning can be trained at scale from raw, unlabeled, imperfect videos by predicting language abstractions instead of pixels, avoiding the cost of generative video prediction.","System-2 cost-minimizing search over VLWM rollouts measurably beats the same model's reactive system-1 decoding (Elo 1261 vs 992), suggesting inference-time reasoning helps even when training demonstrations are imperfect.","The self-supervised critic generalizes beyond its training domain: 92.7% on egocentric goal-achievement detection and 72.9% on open grounded planning robot subsets, outperforming semantic-similarity baselines.","VLWM's interleaved action-state format gives SOTA VPA results with an 8B model, beating a 70B-assisted VidAssist on most metrics, so world-model pretraining is a viable alternative to scaling the planner alone.","Goal interpretation and world-state descriptions are load-bearing for the critic: removing them hurts OOD goal-achievement detection, especially on egocentric data."],"supporting_citations":[{"why":"Defines the VPA benchmark and provides the VLaMP baseline that VLWM must outperform in Table 2.","marker":"(Patel et al., 2023)"},{"why":"Provides the VidAssist baseline, the strongest prior VPA planner using a 70B LLM.","marker":"(Islam et al., 2024)"},{"why":"Supplies the Perception Encoder and PerceptionLM used to build the Tree of Captions from raw video.","marker":"(Bolya et al., 2025)"},{"why":"Supplies the Self-Refine procedure that iteratively refines caption-derived drafts into goal-plan trajectories.","marker":"(Madaan et al., 2023)"},{"why":"Frames the whole system-2 architecture: a reward-free world model that rolls out candidate actions and selects by minimizing a cost.","marker":"(LeCun, 2022)"},{"why":"HowTo100M provides the main large-scale web instruction video source for VLWM pretraining.","marker":"(Miech et al., 2019)"},{"why":"NaturalReasoning supplies 1.1M text-only chain-of-thought examples repurposed into action-state trajectories.","marker":"(Yuan et al., 2025)"},{"why":"Defines RoboVQA, the robotics VQA benchmark on which VLWM reports BLEU scores.","marker":"(Sermanet et al., 2024)"},{"why":"Defines WorldPrediction-PP and the Socratic LLM evaluation protocol used to benchmark the critic zero-shot.","marker":"(Chen et al., 2025)"}],"fun_headline_variants":["8B VLWM predicts future in words, beating 70B planners","Language world model: 8B beats 70B on planning via future-state text","VLWM's system-2 lifts Elo 27% over system-1 in planning","Human raters prefer VLWM 2-to-1 over 70B planners"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The training targets extracted by Llama-4 Self-Refine from Tree of Captions are treated as faithful, goal-directed ground truth — with no external verification — so any hallucinated or commonsense-invented state descriptions are learned as world dynamics.","fun_headline_variants_meta":{"raw":{"variants":["8B VLWM predicts future in words, beating 70B planners","Language world model: 8B beats 70B on planning via future-state text","VLWM's system-2 lifts Elo 27% over system-1 in planning","Human raters prefer VLWM 2-to-1 over 70B planners"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001045,"raw_usage":{"total_tokens":4236,"prompt_tokens":754,"completion_tokens":3482,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":3394}},"tokens_in":498,"tokens_out":3482,"duration_ms":29412,"temperature":1.0,"reasoning_tokens":3394,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:27:19.629315+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take held-out instructional videos, adversarially replace the state-description portion of extracted training targets with descriptions from a different task while keeping actions and visuals intact, then test whether VLWM rollouts and critic cost rankings degrade; if they do not, the state descriptions are not doing the claimed dynamics work. A cheaper check: have humans compare VLWM's predicted world-state text against the actual next visual frames.","supporting_citations":[],"review_version":1}