{"id":"9f34a46a-a9ae-42e6-9ca4-511beaa7cc64","arxiv_id":"2507.18178","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"The accuracy gap between fast and slow thinking is proposed as a measure of reasoning contribution, and is used to show that reasoning is domain-specific, scaling mainly reduces overthinking, and knowledge and reasoning separate across network layers.","lead":"This paper proposes a way to separate how much of an LLM's answer comes from stored knowledge versus step-by-step reasoning, by comparing answers given directly against answers given after chain-of-thought thinking. The method is applied to 15 models and suggests that reasoning helps most in math, physics, and chemistry, and that larger models get 'prudent' by overthinking less.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central measure δ (Eq. 5) is not a pure capability: it is bounded by fast accuracy, so cross-domain and cross-model claims may reflect ceiling effects rather than reasoning adjustment.","rationale":"The reader's weakest assumption identifies the external-judge and prompt confounds, which are real implementation risks. However, the deeper issue is internal to the algebra: the paper defines CRA as the raw accuracy gap δ (Eq. 5) and then uses this gap for all headline claims. Equation 9 shows δ = (rc·|Dfalse_fast| − ro·|Dtrue_fast|)/|D|, so δ is determined jointly by the reasoning rates rc/ro and by the knowledge-determined pool sizes |Dfalse_fast| and |Dtrue_fast|. A domain or model with low fast accuracy mechanically has a larger correction pool and a smaller overthinking pool, making positive δ more likely even if the underlying reasoning adjustment is unchanged. Thus the claimed domain-specific ordering (math/physics/chemistry highest) may simply track the domains where fast thinking is weakest, and the scaling conclusion that knowledge improves more than reasoning may reflect the shrinking error pool rather than a genuine difference in capability growth. The CKA localization claim is less affected by this particular ceiling issue, but it is still downstream of the δ interpretation and has its own token-length confound. Because the proposed δ_norm analysis is straightforward with the released datasets and would directly settle whether the phenomena survive, the appropriate verdict remains conditional rather than acceptance or rejection. I partially agree with the reader: their concern is valid but focuses on an external confound, whereas the ceiling effect is a conceptual confound in the measure itself and should be tested first.","tokens_in":19665,"tokens_out":6170,"duration_ms":66964,"concrete_test":"Re-run the MMLU analysis with a fast-accuracy-controlled reasoning gain. For each model and domain, compute a normalized gain δ_norm = (Aslow − Afast) / max(1 − Afast, ε) for a small ε, or stratify questions by fast-answer correctness so that slow-fast comparisons are made within matched fast-accuracy bins. Then repeat the domain ranking (Table 2, Fig. 2) and the scaling analysis (Figs. 2–3) using δ_norm. If mathematics/physics/chemistry no longer dominate the top ranks, or if the 'knowledge improves more than reasoning' pattern disappears, the paper's conclusions are artifacts of the ceiling effect rather than evidence about reasoning adjustment. As a secondary check, score slow answers by exact-match extraction of the final letter instead of the GLM-4-PLUS judge; if δ values change materially, the evaluation protocol is also a confound.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The framework identifies reasoning adjustment capability CRA with δ = Aslow − Afast (Eq. 5). Because accuracy is bounded by 1, δ ≤ 1 − Afast and δ is mechanically compressed when fast accuracy is high. Since Afast varies across domains and model sizes, all three headline conclusions inherit this base-rate confound. (i) Domain-specificity: knowledge-intensive domains tend to have higher fast accuracy, leaving little room for positive δ and ample room for overthinking losses; math/physics domains have lower fast accuracy, creating large correction opportunities. The observed top ranking of math/physics/chemistry and negative δ in political science/history may be a ceiling effect. (ii) Scaling: as models scale, fast accuracy rises, which automatically shrinks the maximum possible δ and shifts the composition of δc and δo; the conclusion that scaling improves knowledge more than reasoning and mainly reduces overthinking could be a tautology of the measure. (iii) The paper does compute rc and ro (Eq. 8), but the headline CRA and abstract claims are stated in terms of δ. No normalization, control for Afast, or external validation of δ as a reasoning measure is provided. The Limitations section admits the method is simplified, but Equations 3–9 require the interpretation of δ as CRA to hold. The separate evaluation protocol (external judge GLM-4-PLUS for slow answers vs. direct comparison for fast) is an additional confound, but the ceiling effect is more fundamental because it operates even under identical evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 'cognition attribution architecture' inspired by dual-system cognitive theory. It defines two phases of LLM inference, knowledge retrieval (fast thinking) and reasoning adjustment (slow thinking), and measures each through prompted accuracy on multiple-choice QA. The central quantity is the accuracy difference δ = Aslow − Afast (Eq. 5), which the paper identifies as the reasoning adjustment capability CRA. The paper then decomposes δ into a correction component δc and an overthinking component δo (Eq. 9). Experiments are run on 15 LLMs over MMLU, MathQA, and MedQA, with domain-level analyses on MMLU. The reported findings are: reasoning adjustment is domain-specific (benefiting mathematics, physics, and chemistry while harming knowledge-intensive domains); parameter scaling improves knowledge more than reasoning and mainly reduces overthinking; and CKA-based representational similarity indicates that knowledge retrieval is localized in lower layers while reasoning adjustment is localized in higher layers.","tokens_in":20054,"tokens_out":5649,"duration_ms":59343,"significance":"If the identification of δ with reasoning adjustment capability were valid, the framework would provide a cheap, model-agnostic tool for decomposing accuracy gains into knowledge and reasoning components, with potential implications for scaling laws, knowledge editing, and small-model reasoning. The paper's strengths include a transparent experimental setup, a broad span of 15 models across three datasets, the useful descriptive decomposition of δ into correction and overthinking rates, and an explicit acknowledgment of limitations. The algebraic identities in Eqs. (3)-(9) are correct. However, the significance is conditional: the headline conclusions are all phrased in terms of δ as a causal capability, and the paper does not establish that δ isolates reasoning adjustment from confounded differences between the fast and slow conditions. The descriptive accuracy-difference results are still of some value, but the claims about 'knowledge' and 'reasoning' as separable cognitive faculties do not currently follow.","major_comments":[{"comment":"The identification CRA := δ in Eq. (5) is mechanically bounded by the fast-accuracy ceiling: δ ≤ 1 − Afast. Since Afast varies across domains and model sizes, the cross-domain and scaling conclusions may reflect base rates rather than reasoning adjustment. For example, from Table 1, Qwen 1.5B has Afast = 53.9 and Qwen 32B has Afast = 79.5, so the maximum possible positive δ for Qwen 32B is about 20.5% versus about 46.1% for Qwen 1.5B. The same mechanism affects the domain rankings in Table 2, where knowledge-intensive domains tend to have high fast accuracy and thus little room for positive δ, while math and physics have low fast accuracy and ample room for correction gains. Eq. (9) shows this directly: δ = (rc·|Dfalse_fast| − ro·|Dtrue_fast|)/|D|, so δ depends on the sizes of the correct and incorrect fast-answer sets. I request a normalization or a control for Afast, for example reporting δ on accuracy-matched subsets, or reporting confidence intervals on rc and ro, or an external validation of δ against an independent reasoning benchmark.","section":"§3.1 and Appendix D.2"},{"comment":"The fast and slow conditions differ in several dimensions beyond the presence of reasoning: the prompts are worded differently (Figures 5 and 6), slow answers are much longer and consume more tokens (Table 6), and the evaluation protocols differ — fast answers are compared directly to the answer key while slow answers are judged by the external model GLM-4-PLUS using a separate prompt (Figure 7). Any of these differences can produce a nonzero δ, so attributing δ to a 'reasoning adjustment capability' is not identified. I ask the authors to run a control where the same evaluator judges both conditions, and ideally also a condition with a matched-length but non-reasoning slow prompt, to isolate the effect of adding reasoning rather than the effect of changing the prompt, output length, or grader.","section":"§3.1 and Appendix D.2"},{"comment":"There is a circularity in the central claims: CRA is defined as δ in Eq. (5), and then statements such as 'reasoning adjustment is domain-specific' and 'parameter scaling improves knowledge more than reasoning' are restatements of aggregate properties of the measured accuracy difference. The decomposition in Eq. (9) is descriptive and useful, but the abstract and conclusion present δ as a measure of a capability. The Limitations paragraph in Section 5 says the method is 'simplified,' but Equations 3–9 require the interpretation of δ as CRA to support the headline findings. I recommend either validating δ against an external reasoning measure (e.g., correlation with performance on held-out reasoning tasks after controlling for Afast) or rewriting the claims to be explicitly descriptive of accuracy differences under the two prompt conditions.","section":"§2 (Eq. 5) and §5"},{"comment":"The 'cognitive hierarchy' claim that knowledge retrieval resides in lower layers and reasoning adjustment in higher layers is based on CKA between fast and slow activations for only two models (Qwen 7B and LLaMA 8B, Figure 4). The fast and slow conditions have different prompts and generate very different numbers of tokens; even though CKA is computed only on question-part tokens, the prompts themselves differ at the input, so the representations may diverge for reasons unrelated to reasoning, such as instruction-following or response format. No significance tests or additional models are reported. Please include more models, use a minimal prompt difference, and add a control with length-matched outputs or identical prompts under different sampling settings.","section":"§3.6 and Appendix H"},{"comment":"No uncertainty quantification is provided for any of the reported accuracies, δ values, or rates. Many per-domain δ entries in Table 2 are within 1–2 percentage points (e.g., several values between −1 and +1), and the domain sample sizes range from about 100 to several hundred questions (Table 4). The top-3 rankings used to support the domain-specificity claim could be unstable under this noise. I request bootstrap confidence intervals or standard errors for δ, and ideally for δc and δo, so that the reader can judge whether differences of a few percentage points are meaningful.","section":"Tables 1–2"}],"minor_comments":[{"comment":"Please fix the typo 'reasnong-intensive' to 'reasoning-intensive'.","section":"§3.3"},{"comment":"The Conclusion says 'parameter scaling primarily enhances both knowledge retrieval and reasoning adjustment, with reasoning adjustment more significant,' which contradicts the Abstract's statement that 'knowledge improvements are more pronounced.' Please correct the wording to be consistent.","section":"§5"},{"comment":"The table header says 'δ of 19 LLMs' but the paper reports 15 models; please correct the count.","section":"Table 2"},{"comment":"The sentence 'the variation in ro across model sizes is more prominent than that in ro' appears to have a typo; it should likely compare ro and rc.","section":"§3.2"},{"comment":"Equation (2) uses the notation Cknowledge and Creasoning, while Equations (3)–(5) use CKR and CRA; please align the notation.","section":"§2 (Eq. 2)"},{"comment":"The statement that for MathQA 'one wrong option is randomly removed to reduce the choices from five to four' introduces a stochastic element; please specify the random seed or describe how the removal was fixed across runs.","section":"Appendix F.2"},{"comment":"The abstract promises that datasets and code will be publicly released, but no repository link is provided; please include the link in the final version.","section":"Abstract and Appendix"}],"recommendation":"major_revision","confidential_remarks":"The paper has an interesting and transparent empirical core, and the descriptive decomposition into correction and overthinking is a useful lens on fast/slow accuracy differences. The main risk is not the algebra, which is correct, but the interpretive leap from δ to a causal 'reasoning adjustment capability.' If the authors can either validate δ against an external reasoning measure or reposition the claims as descriptive observations about accuracy under two prompt conditions, the paper would be publishable. The confounds from prompt wording, output length, token budget, and the differing evaluation protocols need to be addressed explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you spend time on this. The central coefficient δ (Eq. 5) is not a pure measure of reasoning adjustment: it is bounded above by 1 − fast accuracy, so the domain and scaling patterns partly reflect ceiling effects. And slow answers are graded by an external judge (GLM-4-PLUS) while fast answers are string-matched directly, which alone can manufacture a gap.\n\nThat said, the paper does something genuinely useful. Decomposing the fast–slow gap into correction and overthinking rates (Eqs. 7–9) is neat, the algebra is correct, and running this across 15 models and 13 domains gives a descriptive map of where chain-of-thought helps or hurts. The observation that small models overthink more than they correct is consistent with earlier results and is worth having as a systematic measurement. The CKA layer analysis is suggestive, though not as strong as the paper claims.\n\nSoft spots, in order. First, the ceiling effect is not a minor wrinkle. Since δ = Aslow − Afast and accuracy is at most 1, domains with high fast accuracy have little room for positive δ and plenty of room for negative δ. Math and physics start with low fast accuracy, so large corrections are possible; political science and history start high, so overthinking dominates. The headline claim that reasoning adjustment is domain-specific may be a base-rate artifact. Second, the evaluation mismatch: having GLM-4-PLUS judge slow outputs and exact-match fast outputs introduces a confound that operates even when reasoning content is constant. Third, CRA is defined as δ in Eq. 5, so the scaling conclusions (knowledge improves more, reasoning becomes more prudent) are partly restatements of the measure rather than discoveries about the underlying faculties. The Limitations section admits the framework is simplified, but the interpretation of δ as CRA is load-bearing and unvalidated.\n\nWho is this for? People working on chain-of-thought costs, small-model reasoning, or measurement methodology will find the decomposition a useful descriptive tool; they should not cite the causal claims without further controls. This deserves a serious referee — the empirical sweep is real and the decomposition is worth publishing — but the authors need to address the ceiling effect and the judge confound, or reframe the paper as a descriptive study rather than a cognition-attribution framework.","headline":"The fast-slow accuracy gap is a clever and clean decomposition, but the paper's central interpretation of it as 'reasoning adjustment capability' is undermined by ceiling effects and an inconsistent evaluation protocol.","tokens_in":20448,"tokens_out":2324,"would_cite":false,"duration_ms":24210,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dual-system framework claims that the accuracy gap between slow and fast LLM thinking measures reasoning adjustment, enabling knowledge and reasoning to be decoupled, and uses that gap to report where each lives and how scaling affects…","keywords":["large language models","knowledge retrieval","reasoning adjustment","dual-system theory","chain-of-thought","overthinking","parameter scaling","CKA representation similarity"],"falsifier":"Grade the same slow-thinking answers two ways — with the paper's external-judge prompt and with a direct match to the answer key — while keeping fast and slow prompts as parallel as possible, and compare the resulting $\\delta$ values and their domain rankings. If the sign or ranking changes materially, the gap is partly an evaluation artifact rather than a pure measure of reasoning adjustment.","tokens_in":19455,"feed_emoji":"🧠","tokens_out":7029,"duration_ms":68660,"temperature":0.7,"pith_summary":"Drawing on dual-system cognitive theory, the paper separates an LLM's answer into a fast knowledge-retrieval phase and a slow reasoning-adjustment phase. It defines knowledge retrieval capability as fast-thinking accuracy and reasoning adjustment capability as the accuracy gap $\\delta = A_{\\text{slow}} - A_{\\text{fast}}$ between the two modes. Testing 15 LLMs on MMLU, MathQA, and MedQA, the paper reports that reasoning adjustment helps mathematics, physics, and chemistry, that it can hurt political science and history, and that parameter scaling improves knowledge more than reasoning, mainly by reducing overthinking rather than improving correction. It also reports that layer-activation similarity between the two modes stays high in lower layers and drops in higher layers, which it reads as knowledge residing low and reasoning high.","feed_headline":"Scaling boosts LLM knowledge more than reasoning, study finds","feed_subtitle":"Fifteen models show slow thinking helps math and physics but hurts history; small models overthink rather than correct.","key_machinery":"The central object is the fast/slow cognitive-mode pair and the accuracy gap between them. The load-bearing identity is Equation 9, $\\delta = \\delta_c - \\delta_o = \\frac{1}{|D|}\\left(r_c |D^{\\text{false}}_{\\text{fast}}| - r_o |D^{\\text{true}}_{\\text{fast}}|\\right)$, which converts a behavioral accuracy difference into two rates: correction and overthinking. This identity carries the domain-specificity and scaling arguments. The layer-localization claim is carried by CKA, a similarity score between the same network layer's activations under the two modes, where lower layers stay similar and higher layers diverge.","core_discovery":"The central claim is that the gap between slow and fast thinking, $\\delta$ in Equation 5, is a capability, $C_{\\text{RA}}$, that can be cleanly separated from knowledge retrieval, $C_{\\text{KR}} = A_{\\text{fast}}$. The paper then decomposes $\\delta$ into correction $\\delta_c$ — cases where slow thinking fixes a fast error — and overthinking $\\delta_o$ — cases where slow thinking overturns a fast correct answer — and explains the sign and size of $\\delta$ as the trade-off between those two terms. On this basis it claims three empirical findings: reasoning adjustment is domain-specific; scaling adds more knowledge than reasoning and mostly makes models more prudent rather than more intelligent; and knowledge retrieval and reasoning adjustment occupy lower and higher network layers respectively.","pith_inferences":["Editorial inference: the same $\\delta$ decomposition could be run on open-ended generation, where the evaluator confound is harder to control; one testable prediction is that the domain ranking of $\\delta$ will shift when slow answers are graded by exact-match instead of by an external judge.","Editorial inference: the localization claim implies a testable intervention — perturbing higher layers during slow thinking should change reasoning gain more than perturbing lower layers, while the reverse should hold for factual recall.","Editorial inference: the overthinking rate may respond to inference-time controls, so capping chain-of-thought length or adding a decide-whether-to-revise step could selectively reduce $\\delta_o$ in small models, which would support the noise interpretation.","Editorial inference: the anchoring result suggests a cheap prudence probe — measuring fast and slow accuracy after injecting a wrong hint; the increase in $\\delta$ under anchoring would quantify how much reasoning compensates when retrieval is misled."],"forward_implications":["Slow thinking should be deployed selectively: in knowledge-intensive domains, chain-of-thought can reduce accuracy, so routing questions by domain could improve average performance.","Scaling laws hide a knowledge/reasoning split: larger models gain mostly memorized knowledge, while reasoning gains saturate once overthinking is controlled, so further reasoning gains require targeted training rather than scale alone.","Because overthinking, not weak correction, dominates small-model reasoning failure, small models need prudence-style training or shorter reasoning chains, not simply more parameters.","If knowledge sits in lower layers and reasoning in higher layers, knowledge editing and reasoning interventions target disjoint places, and the paper's CKA curves provide a map for choosing intervention layers."],"supporting_citations":[{"why":"Supplies the dual-system theory behind the two-phase knowledge/reasoning decomposition.","marker":"[23]"},{"why":"Supplies the chain-of-thought prompting protocol used for slow thinking.","marker":"[14]"},{"why":"Provides the main MMLU benchmark and its subject structure.","marker":"[26]"},{"why":"Provides the MedQA domain dataset for medicine.","marker":"[27]"},{"why":"Provides the MathQA domain dataset for mathematics.","marker":"[28]"},{"why":"Supplies the representation-similarity metric used for the layer-localization analysis.","marker":"[36]"},{"why":"Supplies the information-gain view of reasoning used to interpret correction benefit.","marker":"[24]"},{"why":"Supplies the noise-introduction view of reasoning used to interpret overthinking loss.","marker":"[25]"}],"fun_headline_variants":["LLM reasoning gains little from scaling, knowledge does","Slow thinking boosts math but hurts history in LLMs","Where LLMs store knowledge vs reasoning: layer split","Fast vs slow thinking reveals LLM knowledge-reasoning split","Scaling LLMs mostly makes them more prudent, not smarter"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the accuracy gap between slow and fast thinking is caused by reasoning adjustment alone, and not by the two modes' different prompts, output lengths, or the fact that slow answers are graded by an external judge while fast answers are matched directly to the answer key; if any of those confounds contributes to the gap, the paper's conclusions about domain specificity, scaling, and layer localization do not follow.","fun_headline_variants_meta":{"raw":{"variants":["LLM reasoning gains little from scaling, knowledge does","Slow thinking boosts math but hurts history in LLMs","Where LLMs store knowledge vs reasoning: layer split","Fast vs slow thinking reveals LLM knowledge-reasoning split","Scaling LLMs mostly makes them more prudent, not smarter"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000579,"raw_usage":{"total_tokens":2733,"prompt_tokens":957,"completion_tokens":1776,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":1706}},"tokens_in":573,"tokens_out":1776,"duration_ms":12979,"temperature":1.0,"reasoning_tokens":1706,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:16:44.586043+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Grade the same slow-thinking answers two ways — with the paper's external-judge prompt and with a direct match to the answer key — while keeping fast and slow prompts as parallel as possible, and compare the resulting $\\delta$ values and their domain rankings. If the sign or ranking changes materially, the gap is partly an evaluation artifact rather than a pure measure of reasoning adjustment.","supporting_citations":[{"cited_title":"Medical exam question answering with large- scale reading comprehension,","cited_arxiv_id":null,"evidence_quote":"Provides the MedQA domain dataset for medicine."},{"cited_title":"Similarity of neural network representations revisited,","cited_arxiv_id":null,"evidence_quote":"Supplies the representation-similarity metric used for the layer-localization analysis."}],"review_version":1}