{"id":"02b1f5c8-5c31-4ee0-a359-516651ea5c37","arxiv_id":"2502.01591","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A transformer world model agent using a static patch tokenizer, warmup before imagination training, and block teacher forcing reaches 69.66% reward on Craftax-classic, beating DreamerV3 and the human expert figure.","lead":"This paper improves model-based reinforcement learning agents that use transformer world models, adding a warmup phase before training on imagined data, a simpler nearest-neighbor image tokenizer, and a parallel block teacher forcing scheme. The resulting agent beats the previous best methods on Craftax and, for the first time, exceeds the human expert reward on that benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward metric in headline is success-rate average that excludes health rewards, while prior reported numbers (DreamerV3, human) include them; SOTA and superhuman claims may hinge on an inconsistent metric.","rationale":"The reader's weakest assumption (human performance on Crafter may not transfer to Craftax-classic) is valid and acknowledged by the paper's asterisk. Our concern is more load-bearing because it affects every quantitative comparison, not just the human line, and it is not acknowledged. The paper's internal ladder (M1-M5) and ablations are credible, but the absolute claim 'SOTA reward of 69.66%' and 'significantly outperforming DreamerV3, which achieves 53.2%' require comparable reward definitions. Footnote 1 shows the paper uses an achievement-success-rate average that excludes health, while prior values are raw environment rewards including health. This is a concrete, testable discrepancy. If resolved and the ordering holds, the paper should be accepted; if not, the headline claims need revision. Hence the verdict remains CONDITIONAL, not changing the reader's assessment.","tokens_in":27781,"tokens_out":9953,"duration_ms":79562,"concrete_test":"Use the official Craftax-classic evaluation to compute, for the paper's best MBRL agent and for DreamerV3, both (a) the paper's R = (1/N)Σs_i and (b) the mean episode reward including health, expressed as a percentage of 22. If the health-inclusive reward of the paper's agent is below 65.0% (or if DreamerV3's health-inclusive reward is above the paper's), the headline claim fails. As a no-code check, inspect the evaluation scripts from Hafner 2021 / Matthews 2024 to confirm whether the reported 53.2% and 65.0% include health; if they do, the paper must rerun with a unified metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline reward is defined in footnote 1 as R = (1/N) Σ s_i, the arithmetic mean of per-achievement success rates, explicitly 'ignoring minor contributions to the reward based on the health of the agent.' Appendix A.2.2 confirms the authors ignore health rewards during TWM training. However, the comparison numbers in Table 1 and the abstract—DreamerV3's 53.2%, human expert's 65.0%, IRIS 25%, Δ-IRIS 35%—are taken from prior work that reports the actual environment reward, which includes -0.1 per damage and +0.1 per healing. In Craftax-classic the environment reward is the sum of achievement rewards plus these health terms, so the paper's R differs from the raw reward by the health contribution. If the health term is negative and non-negligible (e.g., -2 to -5 percentage points), the claimed 69.66% would drop below the human 65.0% threshold, and the 'surpassing human for the first time' headline would fail. The paper never reports the health component or adjusts prior numbers to a common metric, so the SOTA and superhuman claims are not yet validated against comparable quantities.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes three modifications to transformer-based world models for model-based RL: (a) Dyna with warmup, training the policy on real and imagined data only after the world model has been trained for a number of steps; (b) a patch-based nearest-neighbor tokenizer (NNT) with a static, growing codebook; and (c) block teacher forcing (BTF), a parallel next-timestep token prediction objective with block-causal attention. The authors evaluate these contributions on Craftax-classic and report a reward of 69.66% and a score of 31.77% after 1M environment steps, claiming to surpass DreamerV3 (53.2%) and human expert performance (65.0%) for the first time. They also include ablations, an external evaluation of TWM rollout quality via a symbol extractor, and additional experiments on Craftax-full, MinAtar, and three OpenSpiel games.","tokens_in":28047,"tokens_out":7842,"duration_ms":59574,"significance":"The three proposed components are clearly motivated and the paper is generally well designed: the ladder of improvements is evaluated with 10 seeds and standard errors, each rung is ablated, and the TWM rollouts are assessed with a separately trained symbol classifier on held-out trajectories rather than only by proxy image error. If the reported results are reproducible, the methodological contributions are likely useful to the MBRL community. However, the headline claims of state-of-the-art and superhuman performance rest on a reward metric that excludes health rewards and on human data collected in a different (though near-identical) environment, so those claims are not yet demonstrated on a common basis.","major_comments":[{"comment":"The paper defines its reward as R = (1/N) Σ_i s_i, the arithmetic mean of per-achievement success rates, explicitly \"ignoring minor contributions to the reward based on the health of the agent\" (footnote 1), and Appendix A.2.2 states that the world model uses a binary reward target that ignores damage and healing. However, the comparison numbers in Table 1 and the abstract (DreamerV3 53.2%, human expert 65.0%, IRIS 25%, Δ-IRIS 35%) are taken from prior work that reports the full environment reward, which includes a -0.1 per damage and +0.1 per healing. These are not the same quantity, and the paper never reports the health component or recalibrates the prior results to a common metric; therefore the statements \"new SOTA reward\" and \"exceeding human performance of 65.0% for the first time\" are not supported by the numbers as presented. The authors should report both the achievement-mean and the full environment reward for all methods, or otherwise adjust the comparison to a single consistent metric.","section":"§1, footnote 1; Table 1; §A.2.2"},{"comment":"The human expert reference values (reward 65.0±10.5, score 50.5±6.8) are measured on Crafter, not on Craftax-classic; the asterisk in Table 1 acknowledges this, but the abstract and Section 4.2 state \"for the first time\" surpassing human performance without qualification. Because human performance may differ between Crafter and Craftax-classic, the paper should either measure human play on Craftax-classic or explicitly hedge the claim in the abstract and conclusion. At minimum, the unhedged \"for the first time\" language should be revised to indicate that the comparison is to Crafter-based human data.","section":"§4.2; Table 1"},{"comment":"The in-house DreamerV3 run is reported at 47.18±3.88, about six points below the published 53.2% figure, and the paper does not specify whether this run was scored with the paper's achievement-mean reward (excluding health) or with the raw environment reward. Without this information, and without a breakdown of the health component for this run, the comparison to DreamerV3 is ambiguous even on the same benchmark. Please clarify the evaluation protocol for this row.","section":"Table 1, 'Our DreamerV3' row"}],"minor_comments":[{"comment":"Equation (3) defines LBTF as a log-likelihood, but the text does not explicitly state that this is the training objective; please clarify the relationship between this objective and the block-causal attention mask shown in Figure 2.","section":"§3.6, Eq. (3)"},{"comment":"In the list of TWM losses, the item beginning \"and (3 The third loss\" has a missing opening parenthesis; please fix the typo.","section":"Appendix A.2.2"},{"comment":"Table 6 lists the Craftax-classic buffer size as 48,000, while Table 5 and Section A.3.4 specify a buffer size of 128,000; please reconcile these values, and check whether the Classic/Full columns are swapped.","section":"Appendix E, Table 6"},{"comment":"The bullet list under Section 4.1 repeats \"M5: M4 + BTF. Our best MBRL (slow)\" twice; the second entry should be reformatted to avoid duplication.","section":"§4.1 bullet list"},{"comment":"The caption refers to \"Rollouts v2\" without defining the label; please clarify what v2 indicates.","section":"Figure 6, right panel"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically interesting and the experiments are thorough on the ablation side, but the headline quantitative claims need to be placed on a consistent metric and benchmark before acceptance. The Dyna-with-warmup and BTF ideas are simple but potentially useful; if the authors can report a consistent reward metric and recalibrated human/DreamerV3 baselines, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful empirical study of three practical improvements to tokenized transformer world models, and the Craftax-classic result under its own metric is probably real. The headline claim that it is the first agent to surpass human expert reward is not supported, because the paper's reward metric ignores health while the human and prior numbers include it.\n\nWhat's good: the ladder structure (baseline -> +Dyna -> +patches -> +NNT -> +BTF) is clean, and each rung is ablated with 10 seeds and standard errors. The NNT is a nice idea: a static, growing codebook gives the TWM a stationary target, which is a plausible reason for the gain. BTF is a sensible modification to teacher forcing and also speeds up sampling. The TWM rollout evaluation using a separately trained symbol extractor is careful and shows that M5 is genuinely better at respecting the game dynamics. I also appreciate that they report both reward and score, and that they flag the discrepancy in their DreamerV3 reproduction.\n\nWhere it gets soft: the abstract and Figure 1 say the agent 'exceeds human performance of 65.0%'. That number comes from Crafter, not Craftax-classic, and is the raw environment reward. The paper's R, defined in footnote 1, is the mean success rate over 22 achievements and explicitly ignores health rewards. The prior numbers (DreamerV3 53.2%, human 65.0%, IRIS 25%, etc.) are raw rewards from Crafter. So the comparison mixes two different quantities. If the omitted health component is typically negative, as it is in Crafter, the true raw reward could be several points below 69.66%, possibly under 65%. The paper never reports the raw reward, so the superhuman and even the SOTA claims are not yet validated against comparable numbers. The authors should report both metrics for their agents and, if possible, re-evaluate prior methods on Craftax-classic with the same metric. This is fixable, but it is the difference between a solid benchmark improvement and a headline result.\n\nOther soft spots: no code or data release, which limits independent verification. The patch size is chosen as the oracle 7x7 grid size of Craftax-classic, though they do show 5x5 and 9x9 are competitive. MinAtar and OpenSpiel require extra per-domain hyperparameter changes, so the generality claim is preliminary rather than strong.\n\nWho this is for: people working on tokenized world models and sample-efficient MBRL. It deserves a serious referee, but the revision needs to align the metrics and tone down the superhuman claim.","headline":"Solid empirical ladder for transformer world models, but the 'superhuman' claim rests on comparing a health-free success-rate metric to prior rewards that include health.","tokens_in":28616,"tokens_out":4434,"would_cite":true,"duration_ms":37833,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer world-model agent reaches 69.66% reward on Craftax-classic after 1M steps, topping the 65% human expert mark, via three targeted changes to how the model is tokenized, trained, and used.","keywords":["model-based reinforcement learning","transformer world models","data-efficient RL","Craftax-classic","nearest-neighbor tokenization","block teacher forcing","Dyna with warmup","sample efficiency"],"falsifier":"Run the same human-expert protocol used for Crafter (a small panel of expert players completing 100 episodes) directly on Craftax-classic and average their rewards; if that average exceeds 69.66%, then the claim of being the first agent to surpass human expert reward on this benchmark is falsified, even though the agent-to-agent state of the art could still stand.","tokens_in":27590,"feed_emoji":"🎮","tokens_out":22076,"duration_ms":156796,"temperature":0.7,"pith_summary":"Training a transformer world model to generate imaginary experience for a reinforcement learning policy usually suffers from two problems: the tokenizer's vocabulary keeps changing during learning, and the model must predict long token sequences one by one, which compounds errors. The paper claims both problems have targeted fixes: a static nearest-neighbor tokenizer over image patches keeps the code vocabulary fixed so the world model learns a stationary mapping; block teacher forcing decodes all tokens of the next timestep in parallel using a block-causal attention mask; and a Dyna-with-warmup schedule trains the policy on real experience first, adding imagined rollouts only after the world model has been trained long enough to be useful. On Craftax-classic, these improvements compound, raising reward from 31.93% for the base transformer-world-model agent to 69.66% after 1M environment steps, past the 65% human expert score that the paper says no prior agent had beaten. The same ladder transfers, with modest adaptations, to Craftax-full, MinAtar, and two-player OpenSpiel games, which the paper offers as evidence that the recipe is general rather than benchmark-specific.","feed_headline":"World-model agent hits 69.66% on Craftax-classic, above human 65%","feed_subtitle":"Three targeted fixes to the world model's tokenizer, training, and rollout schedule combine to set the new record.","key_machinery":"The load-bearing pieces are three named mechanisms. Nearest-neighbor tokenizer (NNT): each non-overlapping image patch is mapped to its closest code in a codebook of previously seen patches whenever the Euclidean distance is below a threshold $\\tau=0.75$; otherwise the patch itself is appended as a new code. Because codes are never updated once added, the token targets presented to the transformer world model are stationary, whereas a VQ-VAE codebook keeps shifting during training. Block teacher forcing (BTF): the world model is trained with a block-causal attention mask and supervision that predicts all $L$ tokens of the next timestep in parallel, conditioned only on the history and the action, so the model reasons jointly about an entire future frame before sampling and avoids autoregressive drift; it also roughly halves rollout time. Dyna with warmup: the Proximal Policy Optimization (PPO) policy is updated immediately on real trajectories, and imaginary rollouts from the world model are mixed in only after a warmup period of $T_{\\mathrm{BP}}=200{,}000$ environment steps, so an immature world model cannot corrupt policy learning; starting imagination at step 0 instead drops reward from 67.42% to 33.54%.","core_discovery":"The central claim is that a transformer world model can be turned into a highly data-efficient reinforcement learning engine if three complementary design choices are made on top of a solid model-free baseline: train the policy on both real and imagined trajectories but delay imagination until the model is trained (Dyna with warmup); tokenize image patches with a static nearest-neighbor codebook instead of a continuously updated VQ-VAE (vector-quantized variational autoencoder); and train the world model with block teacher forcing, predicting all tokens of the next frame in parallel. Each rung of the resulting ladder contributes a clear jump in reward on Craftax-classic — from 31.93% (baseline) to 43.36% (Dyna), 58.92% (patch tokenization), 64.96% (nearest-neighbor codebook), and finally 67.42% with block teacher forcing, or 69.66% with additional world-model training. The paper reports this best agent as a new state of the art, surpassing the prior best 53.2% (DreamerV3) and exceeding the 65.0% human expert reward (a statistic carried from the near-replica Crafter benchmark) for the first time, with a score of 31.77% that still trails the human expert score of 50.5%.","pith_inferences":["A direct testable extension is to measure human expert performance on Craftax-classic using the original Crafter protocol. The headline comparison leans on a 65% human number carried over from the near-replica Crafter benchmark, a caveat the paper itself asterisks; if the true human value differs materially, the 'first time above human' framing may not survive even though the agent-to-agent state ","The paper's own rerun of DreamerV3 gives 47.18% rather than the cited 53.2%, which the paper attributes to hyperparameter discrepancies, so the claimed margin over the prior state of the art depends on which number one trusts; an independent rerun of DreamerV3 under the paper's exact protocol would settle the comparison.","The stationary-codebook idea should transfer beyond grid games: any world model whose observation tokenizer keeps shifting forces the dynamics model to chase a moving target. Applying the NNT-style static codebook to continuous-control or Atari environments would separate the benefit of patch alignment, which the paper shows matters (5×5 and 9×9 patches are worse than 7×7), from the benefit of a f","Block teacher forcing's factor-of-two speedup suggests that parallel frame decoding could make imagined rollouts cheap enough for decision-time planners such as tree search, not just background planners, which is an extension the paper does not pursue."],"forward_implications":["Removing any single rung of the ladder hurts: dropping Dyna lowers reward to 55.02%, dropping NNT to 60.66%, dropping both NNT and patches to 45.86%, and dropping BTF to 64.96%.","Starting imagination at step 0 instead of after the warmup collapses reward from 67.42% to 33.54%, making the warmup load-bearing rather than cosmetic.","Block teacher forcing is roughly twice as fast as autoregressive rollout while being more accurate, improving both the reward and the compute budget of imagination.","The improved model-free baseline (a CNN+GRU policy trained with PPO) alone reaches 55.49% on Craftax-classic, above DreamerV3's reported 53.2%, so the MBRL gains stack on top of a competitive model-free agent.","The same ladder of improvements also improves results on MinAtar, Craftax-full, and three two-player OpenSpiel games, supporting generality beyond Craftax-classic."],"supporting_citations":[{"why":"Defines the original Dyna-style mixing of real and imagined experience that this paper revives as Dyna with warmup.","marker":"(Sutton, 1990)"},{"why":"Introduces the VQ-VAE discrete tokenizer that serves as the baseline and is replaced by the nearest-neighbor tokenizer.","marker":"Van Den Oord et al. 2017"},{"why":"Provides the IRIS transformer world model and tokenizer setup used as the base architecture and comparison point.","marker":"(Micheli et al., 2022)"},{"why":"DreamerV3, the main world-model baseline this paper outperforms and whose reported numbers are the prior state of the art.","marker":"(Hafner et al., 2023)"},{"why":"The Crafter benchmark whose human expert reward (65%) is the source of the human-comparison claim.","marker":"(Hafner, 2021)"},{"why":"Defines the Craftax-classic and Craftax environments where all central experiments run.","marker":"(Matthews et al., 2024)"},{"why":"Supplies the model-free baseline agent (PPO with a CNN policy) that this paper's MFRL agent improves upon.","marker":"Moon et al. (2024)"},{"why":"Introduces parallel next-frame token prediction in retentive environment models, which block teacher forcing resembles and extends to any transformer.","marker":"(Cohen et al., 2024)"}],"fun_headline_variants":["World model beats human on Craftax-classic: 69.66%","Craftax-classic: world model first to beat human, 69.66%","Data-efficient world model: three tweaks beat human on Craftax","Three world-model fixes push Craftax score past human: 69.66%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 'first time above human expert' headline assumes that the 65% human reward measured on the original Crafter benchmark applies to the near-replica Craftax-classic environment, an assumption the paper itself flags with an asterisk stating the number 'may not exactly match Craftax-classic.'","fun_headline_variants_meta":{"raw":{"variants":["World model beats human on Craftax-classic: 69.66%","Craftax-classic: world model first to beat human, 69.66%","Data-efficient world model: three tweaks beat human on Craftax","Three world-model fixes push Craftax score past human: 69.66%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001466,"raw_usage":{"total_tokens":5942,"prompt_tokens":1037,"completion_tokens":4905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":4832}},"tokens_in":653,"tokens_out":4905,"duration_ms":33816,"temperature":1.0,"reasoning_tokens":4832,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:53:01.285985+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same human-expert protocol used for Crafter (a small panel of expert players completing 100 episodes) directly on Craftax-classic and average their rewards; if that average exceeds 69.66%, then the claim of being the first agent to surpass human expert reward on this benchmark is falsified, even though the agent-to-agent state of the art could still stand.","supporting_citations":[],"review_version":1}