{"id":"999257e5-517b-44fe-b48e-02562726a56d","arxiv_id":"2507.17699","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"With Python tool augmentation, DeepSeek-R1 and Qwen 3 Thinking outperform their non-thinking counterparts on River Crossing and Blocks World, but the advantage does not appear on Hanoi or Checker Jumping.","lead":"Apple's recent claim that reasoning models do not truly reason is revisited. When the same models are allowed to use Python and a scratchpad, reasoning models beat plain models on several puzzle tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PoT advantage plausibly reflects code-generation skill, not the thinking process, because the paired models differ in multiple uncontrolled ways.","rationale":"The reader's weakest_assumption correctly identifies the paired-model design as the weak point. My independent read of Tables 2 and 3 confirms the concern: the LRM advantage is concentrated in the PoT rows of River Crossing and Blocks World, which are exactly the tasks where generating the right search algorithm in code dominates. On Hanoi Tower, where every model with PoT gets 5/5, no thinking advantage appears, and on Checker Jumping no model succeeds at any N>=3. So the paper's own data show the claim is task- and tool-dependent, not the \"consistently across all levels\" stated in the abstract. The most load-bearing assumption is that the DeepSeek-V3/R1 and Qwen 3/Qwen 3 Thinking pairs are otherwise equivalent except for the thinking mechanism. That assumption is clearly imperfect: R1 and V3 are different releases, and even though both are 37B, the RL training that produced R1 plausibly changed code-generation ability; Qwen 3 Thinking is a mode of the same base model, which is a cleaner comparison but still mixes decoding behavior with any prompting-side effects. The paper provides no auxiliary measurement of code-generation skill or an ablation in which a single model is toggled between reasoning and non-reasoning modes. Thus the central claim is underdetermined, and CONDITIONAL is the right verdict. I differ slightly from the reader only in emphasis: the lack of statistical reporting is secondary; the primary issue is attribution, and the proposed code-probe test would settle it directly. I do not think the paper should be rejected because the narrow result (PoT enables R1 and Qwen Thinking to solve these puzzles while their counterparts fail) is reproducible in principle and the source code is provided; but the overbroad framing and uncontrolled comparison justify requiring the additional test.","tokens_in":15748,"tokens_out":1821,"duration_ms":15900,"concrete_test":"Run a controlled code-generation probe: give DeepSeek-V3, DeepSeek-R1, Qwen 3, and Qwen 3 Thinking identical Python-generation prompts on a separate suite of algorithmic programming problems (e.g., implement BFS for sliding puzzles) with no puzzle-specific reasoning required, and compare success rates under the same PoT harness. If V3 and Qwen 3 match or exceed R1 and Qwen 3 Thinking on such a code-only probe, then the River Crossing/Blocks World gap is not explained by general code skill; if the reasoning models are also better at code generation, the paired-model comparison cannot attribute the gap to thinking without additional controls.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that \"with proper tool use, LRMs consistently outperform their non-reasoning counterparts across all levels of task complexity.\" What Table 3 actually shows is that under Program-of-Thought, DeepSeek-R1 and Qwen 3 Thinking generate executable Python that solves River Crossing and Blocks World at all tested N, while DeepSeek-V3 and Qwen 3 do not. The paper treats each LRM/LLM pair as differing only in \"thinking,\" but DeepSeek-R1 and DeepSeek-V3 are separate model releases with different training, alignment, and presumably different base code-generation abilities, while Qwen 3 Thinking is a decoding mode of Qwen 3, so the pairs are not controlled in the same way. The abstract's \"consistently\" is also contradicted by the paper's own tables: on Hanoi Tower all four models reach 5/5 with PoT, so there is no thinking advantage there, and on Checker Jumping all models score 0/5 for all N>=3. Thus the headline claim depends on interpreting the PoT River Crossing and Blocks World gaps as measuring the value of the thinking process. If the reasoning variants are simply better Python code generators, the observed gap reflects code skill, not reasoning. The paper does not report any control isolating thinking from other model differences, such as matching code-generation capability on held-out programming tasks or ablating thinking within the same model, and with only five runs and no variance reporting the gap is not statistically secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper revisits Apple's 'thinking illusion' benchmark and asks whether Large Reasoning Models (LRMs) gain an advantage over non-reasoning LLMs when both are augmented with external tools. The authors evaluate DeepSeek-V3/R1 and Qwen 3/Qwen 3 Thinking on four puzzles (Hanoi Tower, Checker Jumping, River Crossing, Blocks World) under direct prompting, Program-of-Thought, Think-and-Execute, and a scratchpad-based multi-step protocol. They report that with proper tool use, LRMs consistently outperform their non-reasoning counterparts across all levels of task complexity, and use this to challenge the 'reasoning is an illusion' narrative. The paper also studies scratchpad usage and token consumption for one reasoning model.","tokens_in":16077,"tokens_out":2219,"duration_ms":24419,"significance":"The question addressed is timely and the study has some strengths: the evaluation reuses a controlled external benchmark, the code is publicly released, and the comparison across four tool-use regimes is systematic. If the main claim were supported, the result that Program-of-Thought unlocks LRM performance on River Crossing and Blocks World would be a useful empirical contribution. However, the paper's headline claim is considerably stronger than what the data show: the reported tables contain multiple conditions where LRMs do not outperform LLMs, and the paired-model comparisons do not isolate the contribution of 'thinking' from other model differences. The significance of the work is thus currently limited by overgeneralized conclusions and insufficient experimental controls.","major_comments":[{"comment":"The abstract claims that 'with proper tool use, LRMs consistently outperform their non-reasoning counterparts across all levels of task complexity,' but this is contradicted by the paper's own results. In Table 2, under PoT all four models achieve 5/5 on Hanoi Tower for every N, so there is no LRM advantage on that task; in Checker Jumping all models score 0/5 for N >= 3 under every tool condition. In Table 3, the LRM advantage under PoT appears only on River Crossing and Blocks World. The claim of consistency across all levels and tasks is therefore not supported; the supported claim is narrower and should be restated accordingly.","section":"Abstract and Section 4.2, Tables 2 and 3"},{"comment":"The paired-model design does not isolate 'thinking' from other model differences. DeepSeek-R1 and DeepSeek-V3 are separately trained model releases with different post-training, alignment, and possibly different base code-generation abilities, while Qwen 3 and Qwen 3 Thinking are the same base model under different decoding modes. Treating both pairs as controlled experiments on reasoning is therefore not valid. To support the claim that the thinking process causes the observed tool-use gains, the authors should provide a control that matches the models on code generation or ablates thinking within the same model.","section":"Section 4.1, Tables 2 and 3"},{"comment":"The central PoT result is plausibly explained by code-generation skill rather than reasoning ability, because PoT requires the model to write an executable Python program. The paper does not report any control for programming competence, and the prompt for code generation is taken from a different source (CKK+24) than the original benchmark prompts. Without such a control, the River Crossing and Blocks World gaps under PoT may reflect differences in Python coding capability between the compared models rather than a 'thinking' advantage.","section":"Section 4.2, observation 1, and Table 3"},{"comment":"All experiments use only five runs per condition, and the paper reports only success counts without confidence intervals or significance tests. Several observed gaps (e.g., 4/5 vs. 0/5 in River Crossing under PoT) may be meaningful, but the paper also relies on smaller differences and on equal scores to make broad claims about tool-use effectiveness and model comparisons. Reporting variance or a statistical comparison would be necessary to secure the headline claims.","section":"Section 4.1, 'Parameter Settings' and all result tables"}],"minor_comments":[{"comment":"The 'Thinking Output Tokens' column lists Qwen 3 as 'Yes,' but the text describes Qwen 3 as a non-thinking LLM and Qwen 3 Thinking as its reasoning counterpart; the table entry appears inconsistent with the experiment setup.","section":"Table 1"},{"comment":"The scratchpad panel contains a typo: 'Sctratchpad' should be 'Scratchpad'.","section":"Figure 2"},{"comment":"The statement that tool use helps only when the base model is strong is not quantified or supported by any metric of base-model strength; consider either removing or substantiating this claim.","section":"Section 4.2, observation 3"},{"comment":"The token-consumption figures show only Qwen 3 Thinking and not the other models, while the text discusses tool effects generally; the figures should be clearly labeled as a case study or additional comparisons should be provided.","section":"Figures 5-8"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a question of current interest, but the main claim in the abstract is materially stronger than the evidence presented. I would encourage the editor to require a revised version that narrows the claims to what Tables 2 and 3 actually show, adds a control or at least a clear discussion of the confound between thinking capability and code-generation skill, and reports variance or significance information. If the authors can provide such a revision, the paper could become a useful empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a small empirical study that runs Apple's thinking-illusion puzzles with tool augmentations and reports that, under Program-of-Thought, the reasoning variants sometimes beat their non-reasoning counterparts. The tables are honest, but the abstract oversells the result, and the confound between 'thinking' and code-generation skill is never addressed. Worth a referee's time, but only with substantial revision.\n\nWhat's actually new: applying PoT, Think-and-Execute, and scratchpads to Apple's benchmark and checking whether reasoning models gain more than plain models. That's a legitimate extension. The observation that DeepSeek-R1 with PoT solves River Crossing and Blocks World while DeepSeek-V3 does not is a real empirical data point, and the paper also reports negative results (Checker Jumping remains unsolved) and token consumption, which is useful. Source code is linked, which helps reproducibility, though there's no commit hash and the runs go through APIs.\n\nWhere it falls down: the abstract claims LRMs 'consistently outperform' across all levels of task complexity, but the paper's own tables show a tie on Hanoi and zero for every model on Checker Jumping. The consistent advantage appears only on Blocks World with PoT for both model families, plus River Crossing with PoT for DeepSeek only. With five runs per condition and no variance or significance testing, the headline is fragile. More importantly, the comparison pairs are not controlled: DeepSeek-R1 and V3 are separate model releases with different training, and Qwen 3 Thinking is a decoding mode of Qwen 3. The gap could reflect better code generation rather than the value of 'thinking.' The paper never tests that, and the title makes a causal claim the design can't support.\n\nWho it's for: people working on LRM evaluation and tool-augmented reasoning. As a cautionary example of overclaiming from a small uncontrolled result, it could be useful in a reading group, but I wouldn't cite it as evidence. I'd send it to peer review expecting major revision: align the abstract with the tables, add more seeds or error bars, and ideally include a control that isolates code-generation skill from reasoning.","headline":"A small empirical study with a useful narrow finding on PoT-augmented reasoning models, but the abstract overclaims and the thinking-versus-code-skill confound is never addressed.","tokens_in":16531,"tokens_out":3749,"would_cite":false,"duration_ms":29262,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the apparent failure of reasoning models on controlled puzzles disappears when both models get Python-interpreter and scratchpad tools, and that proper tool use makes reasoning models consistently outperform their…","keywords":["large reasoning models","tool augmentation","Program-of-Thought","scratchpad","thinking-illusion benchmark","River Crossing puzzle","Blocks World puzzle","chain-of-thought"],"falsifier":"Run the same PoT and scratchpad prompts on Qwen 3 with its thinking mode toggled on and off for River Crossing and Blocks World at $N=7,9,11,13$, holding the tool and prompt fixed; if the non-thinking mode matches the thinking mode's accuracy, the paper's paired-model comparison does not isolate the thinking process. The deciding observation is whether the success gap survives when the only thing that changes is the presence of explicit reasoning.","tokens_in":15474,"feed_emoji":"🧩","tokens_out":7184,"duration_ms":71376,"temperature":0.7,"pith_summary":"The paper claims that the 'thinking is an illusion' result — reasoning models appearing no better than ordinary LLMs on controlled puzzles — reverses once both model types are given external tools. The authors evaluate DeepSeek-R1 and Qwen 3 Thinking against DeepSeek-V3 and Qwen 3 on the thinking-illusion benchmark's four puzzles, adding a Python interpreter via Program-of-Thought and a scratchpad. Under Program-of-Thought, the reasoning variants solve River Crossing and Blocks World at every tested problem size, while their non-reasoning counterparts mostly fail; on Tower of Hanoi, PoT makes all models perfect. The paper's central claim is that tool augmentation removes the output-length and bookkeeping limits that hid the reasoning models' advantage, making the earlier negative conclusion an artifact of the evaluation setup rather than a genuine absence of reasoning ability.","feed_headline":"Reasoning models beat plain LLMs once given Python tools","feed_subtitle":"On controlled puzzle tasks, R1 and Qwen 3 Thinking pull ahead of their base models with tool access.","key_machinery":"The machinery is a tool-augmented evaluation protocol built on two tools. Program-of-Thought (PoT) prompts the model to write executable Python code that an external interpreter runs, replacing long serial move lists with a compact program. The scratchpad is an external memory interface: the model outputs a partial answer plus a JSON state each step, with early stopping, so long solutions can be spread over multiple rounds. These tools do the work of removing output-length limits and externalizing bookkeeping, which the paper argues is what was suppressing LRM performance.","core_discovery":"The paper claims that the earlier conclusion that explicit reasoning does not help — the 'thinking illusion' — was drawn under tool-free conditions that hard-limited reasoning models by capping their output length. When the same benchmark puzzles are given to both reasoning and non-reasoning models together with a Python interpreter or a scratchpad, reasoning models such as DeepSeek-R1 and Qwen 3 Thinking no longer look weak: with Program-of-Thought they solve River Crossing and Blocks World for every tested size $N=3,\\dots,13$, while DeepSeek-V3 and Qwen 3 mostly fail on those tasks. On Tower of Hanoi, PoT gives all four models perfect accuracy at every tested $N$, and scratchpad use lifts Blocks World performance substantially. The paper concludes that proper tool use lets LRMs consistently outperform their non-reasoning counterparts, and that the limitation observed in earlier studies was an evaluation artifact rather than a missing reasoning ability.","pith_inferences":["An implication the paper leaves implicit is that the cleanest test of its central claim uses a single model family with thinking as a decoding toggle, so the reasoning process is the only variable; Qwen 3's setup already approximates this.","The results predict that reasoning models will keep failing on tasks like Checker Jumping where the hard part is search rather than bookkeeping, because no compact program removes the combinatorial difficulty.","A practical extension of the finding is that tool-augmented reasoning models can act as dependable small planners in constrained domains, with the interpreter handling state tracking the language model cannot sustain over long horizons."],"forward_implications":["Under Program-of-Thought, DeepSeek-R1 and Qwen 3 Thinking solve River Crossing and Blocks World for every tested size from $N=3$ to $N=13$, while their base counterparts mostly fail or degrade sharply.","On Tower of Hanoi, PoT lifts all four models to perfect accuracy at every tested $N$, including sizes where direct prompting produced zero successes for the non-reasoning models.","Checker Jumping remains unsolved for $N \\geq 3$ across all models and tool frameworks, so tool access helps only with tasks that admit an algorithmic expression.","Among the three tool frameworks, PoT gives the largest gains, followed by scratchpad, while Think-and-Execute gives the smallest, indicating that the tool's design matters as much as its presence.","Tool use does not necessarily increase token consumption: for Qwen 3 Thinking, multi-step tool frameworks reduced both thinking tokens and output tokens on some tasks."],"supporting_citations":[{"why":"supplies the four puzzle environments and the earlier tool-free result that the paper sets out to overturn.","marker":"[SMA+25]"},{"why":"contributes the Program-of-Thought method that produces the paper's main accuracy gains.","marker":"[CMWC23]"},{"why":"contributes the Think-and-Execute framework and the code-generation prompt template adapted for PoT.","marker":"[CKK+24]"},{"why":"defines DeepSeek-R1, the reasoning model whose PoT results anchor the main comparison.","marker":"[GYZ+25]"},{"why":"defines DeepSeek-V3, the non-reasoning counterpart in the paired comparison.","marker":"[LFX+24]"},{"why":"defines Qwen 3 and Qwen 3 Thinking, the second paired comparison in the evaluation.","marker":"[YLY+25]"}],"fun_headline_variants":["Tools show reasoning isn't an illusion","Python and scratchpads unlock reasoning models","Tool-augmented thinking beats plain models on puzzles","Earlier reasoning limits vanish with tool use","With tools, reasoning models beat plain LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paired-model design assumes that each reasoning model and its non-reasoning counterpart are otherwise equivalent, so any gap under tool use measures the value of thinking rather than differences in training, data, or code-generation skill between the two model releases.","fun_headline_variants_meta":{"raw":{"variants":["Tools show reasoning isn't an illusion","Python and scratchpads unlock reasoning models","Tool-augmented thinking beats plain models on puzzles","Earlier reasoning limits vanish with tool use","With tools, reasoning models beat plain LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001139,"raw_usage":{"total_tokens":4717,"prompt_tokens":923,"completion_tokens":3794,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":3727}},"tokens_in":539,"tokens_out":3794,"duration_ms":28342,"temperature":1.0,"reasoning_tokens":3727,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:17:59.244504+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same PoT and scratchpad prompts on Qwen 3 with its thinking mode toggled on and off for River Crossing and Blocks World at $N=7,9,11,13$, holding the tool and prompt fixed; if the non-thinking mode matches the thinking mode's accuracy, the paper's paired-model comparison does not isolate the thinking process. The deciding observation is whether the success gap survives when the only thing that changes is the presence of explicit reasoning.","supporting_citations":[],"review_version":1}