{"id":"d4268f25-240a-413c-a3d7-89d7ed09b317","arxiv_id":"2608.13552","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A benchmark that evaluates video world models through agent-driven long-horizon objectives, showing current models are unreliable at maintaining geometric and temporal consistency.","lead":"PlayWorld is a new benchmark that uses AI \"player\" agents to steer video world models toward long-horizon goals, such as orbiting a landmark or walking into water. It tests nine world models and finds they still struggle with spatial consistency and persistent state evolution over 10 to 60 second interactions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adaptive Agent Player is the unmeasured confound: per-model modification statistics are absent for the full benchmark, so Table 2 scores may reflect agent compensation rather than world-model ability.","rationale":"The paper has genuine strengths: 171 human-curated cases, four evaluation dimensions, a human validation study with positive Spearman correlations, and a robust qualitative finding that long-horizon evolution scores are low across models. The central claim, however, is that the adaptive Agent Player makes cross-model scores comparable. That claim rests on the assumption that the agent's online decisions do not systematically help or hinder different models. The reported ablation is too thin: it covers only 25 cases and two closed-source models, and it reports only an aggregate modification ratio rather than the distribution of decisions or the resulting trajectory states. The issue is compounded by the skipped-case policy in Appendix A, where HY-World2 and Hunyuan-GameCraft-2 are evaluated on different subsets; even a perfect agent cannot make means over different case sets comparable. The proposed test directly checks whether the rankings survive when adaptation is removed or held constant. If they do, the confound is bounded; if not, the benchmark still measures something valuable but not yet the claimed fair comparison. This supports the reader's conditional verdict rather than a change in either direction.","tokens_in":18260,"tokens_out":7547,"duration_ms":81138,"concrete_test":"Release per-case execution records for all nine models (already promised in the repository) and recompute Table 2 on the subset of cases in which the Agent Player made zero adaptive modifications (decision sequence identical to the basic action sequence). If the ranking on this subset differs from Table 2 by more than one rank position for any two models, adaptive compensation is a material confound; if the zero-modification subset is too small, compare instead on the common set of cases that pass trajectory validation under both Preset-Only and Preset+Agent policies.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PlayWorld's central claim is that closed-loop adaptation makes scores comparable across models whose action granularity differs (Sec. 3.1). This requires the Agent Player to steer every model to the same objective state with equal difficulty; otherwise the rubric score measures (model + agent compensation), not the model. The only evidence is Table 4, which reports an Agent-modified Action Ratio of 12.0% and 14.9% on 25 cases for Genie 3 and HappyOyster. That does not bound per-case variation, does not cover the other seven models, and does not show that modified actions reach comparable states. The trajectory-validation gate (Sec. 4.1, Table 3) is applied to the adapted rollout, so a sluggish model can be helped by Extend/Correct while a responsive one is not; pass rates from 14.6% to 93.0% therefore already mix control ability with agent effort. Because invalid rollouts are scored as 1, the final four-dimensional scores inherit this mix. Until per-model decision statistics (Keep/Stop/Extend/Correct/End counts, rollout lengths, modification fraction by case) and a sensitivity analysis are reported, the nine-model comparison is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"PlayWorld is a benchmark for evaluating interactive video world models under long-horizon, scene-grounded objectives. It comprises 171 human-annotated scenarios, each with an initial image, an objective such as 'rotate the camera 360 degrees' or 'walk into the water', a basic action sequence used as a shared reference, and a sample-specific VQA rubric. A multi-modal Agent Player (Claude Haiku 4.5) observes generated frames and adapts execution through Keep/Stop/Extend/Correct/End decisions; Gemini 3.1 Pro then scores each rollout across four dimensions (geometry consistency, interaction fidelity, out-of-sight evolution, insight evolution), after a trajectory-validity gate that assigns the minimum score of 1 to invalid rollouts. Nine world models are compared, with the headline finding that all models score poorly on out-of-sight and insight evolution (Table 2, maxima 1.81 and 1.95). The VQA scores are validated against human preferences (Spearman rho = 0.745-0.983, Appendix D), and basic ability metrics (VBench-style quality and VGGT-based controllability) are reported separately. The paper's central methodological claim is that closed-loop adaptation makes scores comparable across models with different action granularity, where fixed action-conditioned benchmarks conflate trajectory failure with world-model ability.","tokens_in":18512,"tokens_out":19179,"duration_ms":161972,"significance":"If the fairness protocol is substantiated, PlayWorld fills a real gap: existing benchmarks such as WorldScore, WorldMark, WBench, and MemoBench drive models with fixed low-level controls and cannot separate trajectory failure from world-model capability. The benchmark itself is substantial and carefully built: 171 human-curated cases, 820+ rubric questions, 1400+ videos, and a promise of code/data release. Several strengths deserve emphasis: the agent-model sensitivity analysis in Table 5 shows that the choice of agent model has limited effect on trajectory quality; Appendix B reports a second scoring pass with low aggregate variance (0.0112); the VQA scores correlate with human preferences on all four dimensions; and the headline finding, that all nine models are unreliable on long-horizon interactive objectives, is falsifiable and useful for the community. The circularity risk is low: rubrics are human-annotated, the agent and verifier are different models, and the human study is external to the rubric questions.","major_comments":[{"comment":"The central claim that closed-loop adaptation yields comparable execution of the same objective across models (Sec. 3.1) is not yet substantiated by the reported evidence. Table 4 reports Agent-modified Action Ratios of 12.0% and 14.9% for only 25 cases and two models (Genie 3 and HappyOyster); no per-model decision statistics (Keep/Stop/Extend/Correct/End counts), per-case modification distributions, or modification rates for the other seven models are reported. This matters because Sec. 4.1 assigns the minimum score of 1 to any rollout that fails trajectory validation, and validation is applied to the agent-adapted rollout rather than to a common reference trajectory. The Table 2 dimension scores therefore mix rubric-measured capability, trajectory-control ability, and agent compensation in model-specific proportions. The geometry consistency pass rates in Table 3 range from 14.6% to 77.1%; for HY-WorldPlay, whose pass rate is 14.6%, the reported score 1.12 corresponds to a conditional rubric score of roughly 1.82 on the valid minority, so the composite is dominated by validation failure, whereas for Genie 3 the composite is mostly rubric-driven. The skeptic's concern is thus valid as stated. I ask for per-model adaptation statistics over the full benchmark, an analysis of whether validated rollouts reach comparable objective states (e.g., per-case completion-state or final-pose error), and a sensitivity analysis reporting rubric scores conditioned on validated rollouts alongside the gated composite.","section":"§3.1, §4.1–4.2, Tables 2–4"},{"comment":"The human-validation study establishes that the VQA verifier's ranking of the produced videos agrees with human pairwise preferences (Spearman rho = 0.933 Overall, 0.983 geometry, 0.933 interaction, 0.812 out-of-sight, 0.745 insight), but it does not bound the agent-compensation confound, because both the human raters and the verifier watched the same agent-adapted rollouts. Agreement between the two is exactly what one would expect even if the closed-loop controller had compensated unevenly across models, since both evaluate the same videos; the study therefore does not show that different models were tested at comparable objective states. The study is also modest in scale (120 pairwise items, 600 judgments, overall Fleiss' kappa = 0.434), and the weakest correlation occurs on insight evolution, the one dimension without a trajectory-validity gate. A control experiment that varies adaptation intensity on matched cases, or human judgments of whether paired rollouts from different models actually satisfy the stated completion condition, would speak to the protocol-level fairness claim that the current study does not address.","section":"§4.4, Fig. 6, Appendix D"},{"comment":"Two models are not evaluated on the full benchmark: HY-World2 skips cases whose initial images do not satisfy its input requirements, and those cases are excluded from its score aggregation, while Hunyuan-GameCraft-2 cannot always follow the intended WAIT-only protocol for insight evolution. These exclusions are disclosed honestly, but their impact on the cross-model comparison is not quantified, and Tables 2 and 3 present all nine models side by side as if computed over the same set of cases. I ask the authors to report the number of evaluated cases per model and per dimension and to recompute the Overall scores on the common subset of cases, so that ranking differences can be separated from case-coverage differences.","section":"Appendix A, Tables 2–3"},{"comment":"The Action Controllability pass rates inherit the same closed-loop concern: the target trajectory is 'defined by the executed actions, including online adjustments made by the Agent Player' (Sec. 3.4), so the target for each model is itself a product of the controller under test. A model whose commands were more heavily tuned through Extend/Stop/Correct decisions receives a target that is easier to match, and the pass rates are additionally computed only over rollouts with valid VGGT estimates. Because Sec. 4.3 draws conclusions about 'Action Controllability' from these numbers, the paper should report the per-model distribution of executed-action adjustments for the controllability subset, or separate 'the model follows the agent's commands' from 'the interaction reaches the intended objective state'.","section":"§3.4, Table 6"}],"minor_comments":[{"comment":"The Geometry consistency paragraph contains spelling artifacts ('TrajectoryV alidity', 'V alidity'), which appear to be spacing or formatting errors; the manuscript should be proofread for such issues.","section":"§3.3"},{"comment":"The caption of the interaction-fidelity example in Fig. 4 spells 'vender' instead of 'vendor'; please proofread the figure captions.","section":"Fig. 4"},{"comment":"The radar chart in Fig. 1 displays the four rubric dimensions on a 1–3 range and Basic Ability on a 0–1 range, as disclosed in Appendix A; the figure caption should state these ranges explicitly, since the chart's visual impression suggests a common scale.","section":"Fig. 1, Appendix A"},{"comment":"The sentence 'We attribute this limited sensitivity to the relatively constrained decision task...' is presented as an explanation without supporting evidence; either provide supporting measurements (such as the distribution of decision types or decision entropy) or reframe the sentence as a hypothesis.","section":"§4.2"},{"comment":"The appendix appropriately cautions that the two-pass analysis 'is not a precise estimate of scoring uncertainty'; adding the per-question agreement rate between the two passes would be useful, since the aggregate mean variance of 0.0112 could hide systematic disagreements.","section":"Appendix B"},{"comment":"The relation between the 40-step interaction budget and the reported 'approximately 10 to 60 seconds' rollout duration is not explained; a short description of how steps, decision latency, and generated video length relate would clarify the protocol.","section":"§3.1, §4.1"},{"comment":"Subject Consistency is reported as N/A when no dynamic subject is detected, and the controllability pass rates are computed over rollouts with valid estimates; the fraction of N/A or invalid cases per model should be reported so that readers can judge the coverage of the basic-ability metrics.","section":"§3.4, Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is well constructed and the topic is timely; my major-revision recommendation rests on a single load-bearing evidentiary gap (per-model Agent Player compensation statistics for the full benchmark), which is feasible to fill within the manuscript's scope. I would also ask the editor to ensure that the per-model case-coverage numbers for the two partially evaluated models are reported, since the current side-by-side tables imply a common evaluation set. Given that several authors are affiliated with an industrial lab and several evaluated models are commercial systems, an explicit funding and conflict-of-interest statement would be appropriate. Finally, the promised release of code, data, and recorded rollouts will be essential for reproducibility given the use of browser automation on closed-source web interfaces."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nQuick take on arXiv:2608.13552 (PlayWorld). This is a real step forward for evaluating interactive world models. Instead of feeding every model the same fixed action sequence, they use a multi-modal agent that adapts actions toward a shared long-horizon objective (rotate 360°, walk into water). That addresses a genuine weakness in benchmarks like WorldMark and MemoBench, where action-granularity differences make scores hard to compare. The four-dimension VQA rubric (geometry consistency, interaction fidelity, out-of-sight evolution, insight evolution) over 171 human-annotated scenarios is well thought out, and the human-alignment study (Spearman ρ 0.745–0.983) provides a solid external check. They also release code and data, which is more than many benchmark papers do.\n\nThe main soft spot is exactly what the stress-test flags: the central fairness claim is under-supported. The ablation in Sec. 4.2 covers only 25 cases and two models, and per-model agent modification statistics (Keep/Stop/Extend/Correct/End counts, rollout lengths, modification ratios) are not reported for the full nine-model run. Without those, we can't bound how much the agent is compensating for sluggish or unresponsive models. Since trajectory validation is applied after adaptation and invalid rollouts get a score of 1, the final numbers mix raw controllability with whatever help the agent provided. This isn't fatal – the protocol is still meaningfully better than preset-only trajectories – but the nine-model ranking should be read as provisional until sensitivity analysis is shown.\n\nTwo smaller issues: HY-World2 and Hunyuan-GameCraft-2 are evaluated on subsets (some cases skipped for input-format reasons), which makes their overall scores slightly apples-to-oranges; and verifier variance is acknowledged but only checked with a single extra pass, not full error bars.\n\nThat said, this deserves peer review. The benchmark artifact is valuable, the idea is right, and the gaps are fixable with more transparency rather than being conceptual problems. If I were editing, I'd send it out with a request for per-model agent statistics and a broader sensitivity analysis.\n\nBest.","headline":"PlayWorld is a genuinely useful benchmark for interactive world models, but the central fairness claim about its adaptive agent-player protocol needs more evidence than the paper currently provides.","tokens_in":19052,"tokens_out":3951,"would_cite":true,"duration_ms":37479,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agent players expose world models' long-horizon failures","keywords":["world models","video generation","agent player","long-horizon evaluation","interactive benchmark","geometry consistency","state evolution","VQA rubric"],"falsifier":"Run the PlayWorld protocol on a single model with two different agent models that have different action-adjustment styles (e.g., conservative vs. aggressive correction) and check whether the model's dimension scores change significantly; if they do, the benchmark is sensitive to agent behavior rather than isolating world-model ability.","tokens_in":1382,"feed_emoji":"🎮","tokens_out":2012,"duration_ms":26347,"temperature":0.7,"pith_summary":"PlayWorld is a benchmark that evaluates video world models the way a human player actually would: by giving each model a long-horizon objective, such as turning the camera 360 degrees or walking into water, and letting an AI agent adapt its actions online to achieve that objective. The paper argues that fixed action sequences, the standard evaluation method, produce incomparable scores across models because the same controls produce different effects in different models. Using 171 scenarios, a VQA rubric verifier, and nine state-of-the-art models, PlayWorld finds that current models remain unreliable on long-horizon interactive objectives, especially on maintaining spatial consistency and persistent state evolution.","feed_headline":"Agent players expose world models' long-horizon failures","feed_subtitle":"A new 171-scenario benchmark finds current models struggle with spatial consistency and state evolution.","key_machinery":"The key mechanism is the Agent Player, a closed-loop control system that observes a model's generated frames and action history and decides to keep, stop, extend, correct, or end the current action. This adapts execution to each model's action granularity, ensuring that all models are evaluated on the same objective rather than on identical—but possibly ineffective—action sequences. The scaffold also includes a VQA rubric verifier that answers sample-specific Yes/No questions after a trajectory-validity gate, converting answers into dimension scores on a 1–5 scale.","core_discovery":"The central claim is that current video world models, though capable of short, visually plausible outputs, fail to sustain coherent worlds over long-horizon interaction. The paper demonstrates this failure through a benchmark that uses a multi-modal Agent Player to adapt actions toward a shared objective, and a VQA rubric that scores four capabilities: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution. Across nine models, the scores are low, with out-of-sight evolution and insight evolution markedly worse, indicating a systemic inability to maintain persistent, causally consistent state changes over time.","pith_inferences":["If the VQA verifier aligns with human preference, a fully automated objective-driven benchmark could replace expensive human evaluation, but the robustness of the verifier across different objectives and models remains a key extension to test.","The finding that geometry consistency breaks down on revisitation suggests that world models lack a persistent global representation of the 3D world; training with explicit memory or 3D anchoring could be a direct extension.","The Agent Player's online adaptation might be more effective than a fixed trajectory, but it introduces the risk of confounding model ability with agent behavior; bounding this for the full nine-model set would be a logical next step."],"forward_implications":["A benchmark with 171 scenarios and nine models showing low scores, especially in out-of-sight and insight evolution, suggests a concrete, measurable capability gap in current world models.","The approach invites model developers to target persistent state evolution explicitly, since current models fail to sustain changes during occlusion or long observation.","The finding that trajectory control can diverge from world-model ability implies that interaction fidelity and state evolution are distinct, separate axes of evaluation.","The paper's claim that fixed-action benchmarks conflate trajectory failure with world-model ability would, if true, invalidate a common evaluation practice in the field."],"supporting_citations":[{"why":"Supplies the video quality metrics used for basic ability evaluation.","marker":"VBench (Huang et al., 2024)"},{"why":"Defines translation and rotation pass rates used for action controllability.","marker":"WorldMark (Xu et al., 2026b)"},{"why":"Provides the depth stability metric and motivates trajectory-validation gating.","marker":"MemoBench (Chen et al., 2026b)"},{"why":"Supplies the temporal consistency metric and serves as a comparison for interaction-centric evaluation.","marker":"Omni-WorldBench (Wu et al., 2026)"},{"why":"Baseline for geometry consistency evaluation that uses fixed trajectories.","marker":"WorldScore (Duan et al., 2025)"},{"why":"One of the evaluated closed-source models; top performer on most dimensions.","marker":"Genie 3 (Google DeepMind, 2025)"}],"fun_headline_variants":["World models crumble under long-horizon agent tests","Benchmark of 171 scenarios reveals world model limits","Agent players prove world models lose coherence over time","Long-horizon interaction exposes world model flaws","World models fail at persistent state evolution"],"cache_read_input_tokens":21248,"weakest_assumption_plain":"The Agent Player's online adaptation produces comparable execution of the same objective across models; if it compensates for one model more than another, the rubric scores may reflect the agent's behavior rather than the model's true ability.","fun_headline_variants_meta":{"raw":{"variants":["World models crumble under long-horizon agent tests","Benchmark of 171 scenarios reveals world model limits","Agent players prove world models lose coherence over time","Long-horizon interaction exposes world model flaws","World models fail at persistent state evolution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1325,"prompt_tokens":887,"completion_tokens":438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":503,"tokens_out":438,"duration_ms":4909,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:17:19.657740+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the PlayWorld protocol on a single model with two different agent models that have different action-adjustment styles (e.g., conservative vs. aggressive correction) and check whether the model's dimension scores change significantly; if they do, the benchmark is sensitive to agent behavior rather than isolating world-model ability.","supporting_citations":[],"review_version":1}