{"id":"3b77c1fa-9e58-46d4-9b6e-84e0af888277","arxiv_id":"2505.13426","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A 7B vision-language model trained with reinforcement learning in a new four-game environment, VLM-Gym, outperforms larger proprietary models and improves its perception and reasoning abilities together.","lead":"This paper introduces VLM-Gym, four simple computer games used to train a 7-billion-parameter vision-language model with reinforcement learning, and shows the trained model beats larger proprietary models including Claude at these games. It also reports that the model's ability to describe the board and its ability to reason about moves improve together during training, a pattern the authors call mutual bootstrapping.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The bootstrapping claim rests on an untested premise that game reward is informative about perception; the paper's own G0-2048 collapse is the regime where that premise fails, so a reward-informativeness test is needed before the title's mechanism can be credited.","rationale":"The reader's weakest_assumption matches the most load-bearing point: the mutual-bootstrapping claim is the scientific novelty, and its causal interpretation rests on reward informativeness. This concern is not about external consensus; it is about internal evidence. The paper honestly reports the G0-2048 failure and the perception-reward ablation, but treats them as boundary cases rather than as tests of the mechanism. A cheap simulation can settle whether the stated premise actually distinguishes the successful games from 2048, and an action-space manipulation would test whether the paper's preferred explanation is causal. I keep the reader's CONDITIONAL verdict because the engineering contribution—VLM-Gym, parallel GRPO, released code—is real and the main model comparisons may survive, but the title's bootstrapping claim needs this test or a softened claim. Since the reader already conditioned on this concern, no verdict change is needed.","tokens_in":23068,"tokens_out":9920,"duration_ms":105440,"concrete_test":"Measure reward informativeness directly: sample 10,000 random states per game and compare P(positive game reward) for a correct-perception policy that picks a legal match or merge versus a wrong-perception policy acting on a shuffled board with random tile identities or coordinates. The Sec. 4.1 explanation requires the wrong-perception reward rate to be near chance in Shisen-Sho and Shisen-Sho-Cifar10 and substantially elevated in 2048; if the gap does not match that pattern, the large-action-space premise fails. As a complementary check, run G0-2048 with an expanded 16-action space (tile coordinate plus slide direction) under identical GRPO; emergence of localization patterns would confirm that action-space size controls bootstrapping.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's title-level claim—that perception and reasoning abilities mutually bootstrap through RL—depends on the premise that the per-step game reward is informative about perception quality, so that wrong perception cannot be rewarded. The authors assert this as 'a large action space that prevents incorrect perception and reasoning process from gaining reward is critical' (Sec. 4.1), but the only evidence is a comparison across games: localization patterns emerge in Shisen-Sho and Shisen-Sho-Cifar10 and precede reward gains (Fig. 5), while 2048 collapses because four actions let wrong perception receive positive rewards (Eq. 1, Fig. 10). This is correlational and the premise is never varied or measured. The paper's own perception-reward ablation (Sec. 4.3, Fig. 8) shows that explicitly rewarding perception accelerates P_acc but does not improve game reward, which is hard to reconcile with the mutual-bootstrapping mechanism: if correct perception is not needed for outcome reward in Shisen-Sho, outcome-RL should not have to incentivize it. Unless reward informativeness is quantified or action-space size is manipulated, the bootstrapping conclusion is an interpretation of these specific games, not a demonstrated general property.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VLM-Gym, a suite of four visual games (2048, Shisen-Sho, Shisen-Sho-Cifar10, Swap) with unified interfaces, parallel action sampling, and adjustable difficulty, intended for GRPO-style reinforcement learning of vision-language models. The authors train Qwen2.5-VL-7B in two regimes: G0 via RL from game rewards with random-step exploration, and G1 via a cold-start SFT on 1,000 teacher (Claude-3.7-Sonnet-Thinking) responses that are prompted with ground-truth perception, followed by GRPO. They report that G1 surpasses G0, the teacher, and several proprietary baselines on all games, and they analyze perception accuracy (P_acc) and reasoning accuracy (R_acc) to argue that perception and reasoning abilities mutually bootstrap during RL training.","tokens_in":23163,"tokens_out":6772,"duration_ms":68793,"significance":"If the reported results are reproducible, the paper makes a useful contribution: it provides a lightweight, parallelizable RL benchmark for VLMs, shows that outcome-reward RL can induce structured perception outputs (localization patterns) and reasoning patterns in a small open VLM, and reports the non-obvious claim that a 7B model with a privileged cold start can beat frontier proprietary models on these synthetic games. Strengths include the open-source release of VLM-Gym and training code, the use of external game scores as rewards (which avoids circular reward design), and the explicit P_acc/R_acc decomposition for studying training dynamics. The significance is real but conditional on resolving the statistical, confounding, and mechanism-evidence issues below.","major_comments":[{"comment":"The headline claim that G1 'consistently surpasses' the teacher is not supported with statistical evidence. Table 1 reports one mean per model per game, with no standard deviation, confidence interval, or number of training seeds, and the text does not specify how many independent runs contribute to each mean. The note explains '100 steps×10' but not '1 step×100' used for Swap, and a single-step protocol is inconsistent with the stated multi-turn evaluation setting in Sec. 3.4. Please report means with variance or bootstrap confidence intervals over at least 5 independent evaluation runs (ideally multiple training seeds) and use a consistent multi-turn protocol for all games, or explicitly justify why Swap is evaluated differently.","section":"Table 1 / Sec. 3.4"},{"comment":"The comparison between G1 and Claude-3.7-Sonnet-Thinking is confounded by privileged perception. The cold-start data are generated by prompting Claude with the environment's ground-truth perception, while G1 is evaluated without such privileged information; the teacher is therefore at a perceptual disadvantage at test time. To support 'surpass their teacher across all games,' the authors should either evaluate Claude with the same ground-truth-perception prompts at test time, or train a cold-start variant that uses the teacher's own non-privileged perception outputs, so that the comparison isolates model capability rather than access to ground truth.","section":"Sec. 3.2 / Table 1"},{"comment":"The central 'mutual bootstrapping' claim rests on an untested premise: that per-step game reward is informative about perception quality, so incorrect perception cannot easily obtain positive reward. The 2048 result (Sec. 4.1, Fig. 10) is exactly the regime where this premise fails, and it is presented as post-hoc evidence rather than a controlled test. The paper should either manipulate action-space size or reward informativeness across otherwise matched games, or quantify per-state the association between perception correctness and reward (e.g., mutual information between P_acc and positive reward), and it should support the 'precedes' claim in Fig. 5 with a statistical test across seeds rather than a single rollout count. In addition, Sec. 4.3 (Fig. 8) shows that an explicit perception reward accelerates P_acc without improving game reward; the paper should explain how this observation is consistent with the claim that outcome reward drives perception improvement through a bootstrapping mechanism.","section":"Sec. 4.1 / Figs. 5, 8, 10 / Eq. (1)"},{"comment":"The GRPO advantage in Eq. (6) normalizes group rewards by their standard deviation. With group size G=5 and rewards that are sums of a few binary components (game reward, format reward, perception reward), many groups will have zero variance, making the update undefined unless zero-variance groups are skipped or a variance floor is introduced. This detail is not stated and is material to reproducibility. Please specify the exact handling of zero-variance groups in the implementation.","section":"Sec. 3.1 / Eq. (6)"},{"comment":"The design does not isolate the effect of the ground-truth-perception cold start from the effect of SFT in general. The cold start simultaneously teaches the output format, game mechanics, action syntax, and perception content, so the observed improvements in G1 relative to G0 could be due to any of these factors. An ablation with a cold start on the same 1,000 states but without the ground-truth perception field (for example, using teacher-generated perception instead) is needed to support the claim that 'perception-enhanced cold start' specifically addresses the inaccurate-credit and sparse-reward problems.","section":"Sec. 3.2 / Sec. 4.3"}],"minor_comments":[{"comment":"The word 'boarder' should be 'broader', and the phrase 'knowing-doing' gap is used repeatedly; please standardize the terminology.","section":"Sec. 1"},{"comment":"The y-axis and the regex rule used to count 'localization patterns' are not defined; please specify what counts as a localization pattern and report variance across runs.","section":"Figure 5"},{"comment":"It is not stated whether the Format Reward and Perception Reward are included in R(q_s, o_i) when computing the group advantage in Eq. (6); please clarify the reward composition used for advantage estimation.","section":"Sec. 3.1 / Eq. (4)"},{"comment":"Figure 9 shows ground-truth perception in the distillation prompt, but G1 at test time receives only the screenshot; please state explicitly that this asymmetry exists and discuss any distribution shift it may cause.","section":"Sec. 3.2 / Fig. 9"},{"comment":"The phrase 'existed experiments' should read 'existing experiments'.","section":"Sec. 4.4"},{"comment":"In Section 1, reference [1] (the Claude 3.7 system card) is cited for the claim about Pokémon, which appears to be the wrong source.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the empirical headline claims would be substantially strengthened by repeated-seed numbers and by a controlled comparison that does not give the cold-start teacher privileged perception. Given the paper's affiliation with a frontier-model developer, the comparison to Claude should be held to a strict standard of fairness. I would not reject on the current evidence, but the paper should not be accepted without the additional experiments identified in major comments 1-3."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious referee, but the title-level claim is the part that needs the most scrutiny. What is new: VLM-Gym is a real contribution — parallel execution and parallel action sampling for GRPO across multiple games, which is genuinely missing from most Gym-style wrappers. The G1 cold start, where the teacher gets ground-truth perception annotations and the student is SFT on those responses before RL, is a clever way to inject a perception prior. The emergence analysis showing localization patterns appearing before reward gains in Shisen-Sho is nice, and the G0-2048 failure case is honestly presented: the model collapses when four actions allow wrong perception to still receive positive reward. Released code and environment mean the claims are checkable.\n\nSoft spots, in proportion. Table 1 is point estimates with no error bars or seeds, so \"consistently surpasses\" is too strong. The Swap evaluation protocol labeled \"1 step x 100\" contradicts the stated multi-turn setting; either the evaluation is single-step or the table is mislabeled. The G1-vs-teacher comparison is not apples-to-apples: the student received ground-truth perception during SFT while the teacher did not at evaluation time. That does not kill the result, but it needs disclosure. The training reward (~0.8/step) versus evaluation score (12.8 over 36 steps) implies a state-distribution shift that is never quantified.\n\nThe biggest issue is the bootstrapping claim. The paper asserts that perception and reasoning mutually bootstrap through RL because a large action space prevents wrong perception from being rewarded. That premise is never tested — action-space size is never varied, and reward informativeness is not measured. The authors' own G0-2048 collapse is the regime where the premise fails, which supports the point but is correlational. More tellingly, the perception-reward ablation (Sec. 4.3, Fig. 8) shows that explicitly rewarding perception speeds P_acc but does not improve game reward. If correct perception is not needed for game reward in Shisen-Sho, then outcome-only RL should not have to incentivize it, which undercuts the mutual-bootstrapping mechanism. The honest conclusion is: the co-evolution is real in these games, but the mechanism is not established.\n\nWho is this for: anyone working on RL for VLMs in interactive environments. The environment and recipe are reusable, and the failure analyses are useful. I would not cite the bootstrapping claim as established fact, but I would cite VLM-Gym and the cold-start recipe. It deserves peer review, not a desk reject. The revisions that would move it from conditional to solid: add seeds and error bars, fix or relabel the Swap protocol, disclose the perception privilege, quantify the train/eval shift, add an SFT-only baseline, and either test reward informativeness (e.g., vary action-space size) or soften the bootstrapping language.","headline":"A genuinely useful VLM RL testbed and a clever perception-enhanced cold start, but the headline 'mutual bootstrapping' claim is an interpretation that the paper's own ablation partially undermines.","tokens_in":23904,"tokens_out":1945,"would_cite":true,"duration_ms":19893,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning with a perception-enhanced cold start lets a 7-billion-parameter vision-language model beat its teacher and frontier proprietary models on four visual games.","keywords":["vision-language models","reinforcement learning","visual games","GRPO","perception and reasoning bootstrapping","knowledge distillation","VLM-Gym","interactive decision-making"],"falsifier":"Reproduce the G0 Shisen-Sho run with a small action space (four macro moves instead of full coordinate pairs) or with rewards that sometimes give points for wrong perception; if localization patterns no longer appear before reward gains and scores stop improving, the bootstrapping claim is an artifact of this game set rather than a general property of VLM reinforcement learning.","tokens_in":22696,"feed_emoji":"🎮","tokens_out":8342,"duration_ms":76439,"temperature":0.7,"pith_summary":"The paper sets out to close the 'knowing-doing' gap in vision-language models (VLMs) by training them to play visual games with reinforcement learning. It introduces VLM-Gym, a suite of four visual games with unified interfaces and adjustable difficulty, and trains a 7-billion-parameter VLM in two ways: G0 by pure RL self-evolution, and G1 after a perception-enhanced cold start of 1,000 teacher-distilled responses that include ground-truth perception. The central claim is that G1 consistently outperforms its teacher and strong proprietary models on all four games, and that perception and reasoning abilities bootstrap each other during RL, with structured localization patterns appearing in rollouts before game rewards rise. If that is right, a modest open model with access to privileged perception data plus outcome-reward RL can beat frontier proprietary agents on these games, and reward-driven training alone can induce structured perception and reasoning.","feed_headline":"7B VLM plus RL outplays leading proprietary models on games","feed_subtitle":"Ground-truth perception injected before RL lets a small 7B model beat its teacher on all four games.","key_machinery":"The load-bearing machinery has two parts. The first is VLM-Gym itself: a set of four visual games (2048, Shisen-Sho, Shisen-Sho with CIFAR-10 tiles, and Swap) built on a unified observation/action interface, with ground-truth perception available from the environment, adjustable compositional difficulty, and parallel action sampling so that GRPO's group-relative advantages can be computed. The second is the training recipe: a perception-enhanced cold start—supervised fine-tuning on teacher-distilled responses in which the prompt contains the ground-truth board description—followed by GRPO with a final reward that is the sum of game reward, a format reward, and an optional perception reward. The paper also uses the emergence of two textual patterns, localization and enumeration, as measurable evidence that perception and reasoning are improving in the model's own rollouts.","core_discovery":"At the center of the paper is a claim about how a vision-language model learns to act: when a VLM is placed in an interactive visual environment and optimized with GRPO on a per-step game reward, its perception and reasoning abilities improve together, each enabling the other. In G0, pure RL produces an 'aha' moment in Shisen-Sho: the model learns to localize every tile with coordinates, and this localization pattern appears before game reward rises. In G1, the authors add a perception-enhanced cold start—1,000 teacher-distilled responses that include ground-truth perception together with the teacher's thinking and actions—and then run the same RL. The result is that the G1-7B model surpasses the teacher and proprietary models on all games (for example, 1070 versus 892 in 2048, 17.5 versus 15.3 in Shisen-Sho, 14.1 versus 8.7 in Shisen-Sho-Cifar10, and 0.78 versus 0.43 in Swap). The paper's central discovery is the bootstrapping loop: correct perception is rewarded only when reasoning converts it into a valid action, and correct reasoning is impossible without accurate perception, so RL pressure improves both together.","pith_inferences":["The bootstrapping mechanism is likely sensitive to action-space size and reward informativeness: the paper's own 2048 result shows the mechanism fails when four actions let wrong perception still score, so the claim should be read as a property of tasks where perception is the bottleneck, not of VLM RL in general.","If the recipe generalizes beyond synthetic games, the same combination—outcome reward plus a cold start that injects ground-truth perception from a privileged simulator—would transfer to embodied tasks where instrumentation supplies perception labels; the paper does not test this transfer.","A direct next experiment would be to add a perception process reward to Swap, since the paper's Section 4.3 shows perception reward accelerates perception accuracy without changing game reward, suggesting it may not be enough to overcome sparse reward on its own."],"forward_implications":["A 7-billion-parameter open VLM can outperform its teacher and leading proprietary models on all four VLM-Gym games when perception-enhanced cold start is followed by GRPO.","Pure reward-driven RL can induce structured perception and reasoning in Shisen-Sho and its CIFAR-10 variant, with localization patterns appearing before reward gains.","The cold start removes the perception prior gap and addresses both the inaccurate reward credit problem seen in 2048 and the sparse reward problem seen in Swap.","Perception and reasoning co-evolve under a final verifiable reward; adding an explicit perception process reward speeds perception accuracy but does not improve final game reward.","VLM-Gym's unified, parallel, compositionally difficult environment makes scalable multi-game RL with GRPO practical for vision-language models."],"supporting_citations":[{"why":"Supplies the GRPO objective and group-normalized advantage estimator used in all RL runs.","marker":"[24]"},{"why":"Establishes the verifiable-reward RL paradigm for reasoning models that this paper transfers to VLMs.","marker":"[8]"},{"why":"Documents Qwen2.5-VL, the base model for both G0 and G1 series.","marker":"[2]"},{"why":"Provides the CIFAR-10 images that make the Shisen-Sho variant perception-hard.","marker":"[14]"},{"why":"Documents the knowing-doing gap and game benchmarks that motivate the environment.","marker":"[21]"},{"why":"System card of the teacher model used for cold-start distillation and as the strongest baseline.","marker":"[1]"},{"why":"Defines the Gymnasium interface standard that VLM-Gym follows.","marker":"[33]"},{"why":"Prior demonstration of GRPO on a vision-language model that the paper extends to interactive games.","marker":"[4]"}],"fun_headline_variants":["RL bootstraps perception and reasoning in 7B VLM","Small VLM plus RL outplays Claude-3.7 on games","Bootstrapping loop: VLM learns to perceive and reason via RL","G1: RL with perception cold start beats proprietary models","Perception and reasoning mutually boost in VLM via RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-step game reward is informative about perception quality, so a model cannot earn positive rewards while perceiving incorrectly; the paper's own 2048 result shows this premise fails when the action space is small.","fun_headline_variants_meta":{"raw":{"variants":["RL bootstraps perception and reasoning in 7B VLM","Small VLM plus RL outplays Claude-3.7 on games","Bootstrapping loop: VLM learns to perceive and reason via RL","G1: RL with perception cold start beats proprietary models","Perception and reasoning mutually boost in VLM via RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000706,"raw_usage":{"total_tokens":3221,"prompt_tokens":1024,"completion_tokens":2197,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":2108}},"tokens_in":640,"tokens_out":2197,"duration_ms":14469,"temperature":1.0,"reasoning_tokens":2108,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:16:28.775221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the G0 Shisen-Sho run with a small action space (four macro moves instead of full coordinate pairs) or with rewards that sometimes give points for wrong perception; if localization patterns no longer appear before reward gains and scores stop improving, the bootstrapping claim is an artifact of this game set rather than a general property of VLM reinforcement learning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the verifiable-reward RL paradigm for reasoning models that this paper transfers to VLMs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"System card of the teacher model used for cold-start distillation and as the strongest baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Gymnasium interface standard that VLM-Gym follows."},{"cited_title":"R1-v: Reinforcing super generalization ability in vision-language models with less than $3","cited_arxiv_id":null,"evidence_quote":"Prior demonstration of GRPO on a vision-language model that the paper extends to interactive games."}],"review_version":1}