{"id":"929aa59b-837d-427c-bbe4-a39353c26c95","arxiv_id":"2507.22928","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper reports that chain-of-thought features extracted by sparse autoencoders and transferred through activation patching improve answer confidence in Pythia-2.8B but not in Pythia-70M, implying a scale threshold for CoT faithfulness.","lead":"This paper asks whether the step-by-step reasoning text a language model writes is actually connected to its internal computation. By comparing a large and a small model, it finds signs of such a connection only in the large model, suggesting that model size matters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Separately trained CoT/NoCoT SAE dictionaries are never aligned, so feature-index patching is semantically undefined and cannot support the causal claim.","rationale":"Read in good faith, the paper attempts a feature-level causal test of CoT faithfulness: if CoT-elicited sparse features carry reasoning, patching them into a noCoT run should raise answer probability. For that test to be meaningful, the 'features' must live in a common representational space. The Method section explicitly trains two separate SAEs, but every subsequent operation treats their indices as comparable: |hCoT - hNoCoT|, hpatch[S] from hCoT and hpatch[not S] from hNoCoT, then decode. Without an alignment step, this is not a swap of semantically matched features; it is a coordinate-wise mixing of two independently learned bases. The paper does not report any verification that corresponding indices encode similar content. The appendix repeats the procedure without addressing this. Hence the 2.8B versus 70M difference and the random-over-top-K effect are not evidence about CoT features. The prompt confound (three few-shot examples versus none) further muddies the CoT/NoCoT comparison, but the dictionary misalignment is more fundamental because it invalidates the causal intervention itself. The reader's weakest_assumption identifies the same issue. I agree with REJECT: the central claim is unsupported as written. A shared-dictionary re-run would be the decisive check; if the effect persists, the conclusion could be rehabilitated, but the current manuscript does not support it.","tokens_in":17725,"tokens_out":4936,"duration_ms":48280,"concrete_test":"Train a single shared SAE on the union of CoT and NoCoT residual activations from Pythia-2.8B layer 2 final token, and re-run the Top-K and Random-K patching experiments in that shared feature space. If the CoT-to-NoCoT log-probability gain at K=2-8 does not replicate, the original scale-dependent causal effect is an artifact of crossing unaligned dictionaries. As a cheap supporting check, compute pairwise cosine similarities between D_CoT and D_NoCoT decoder columns and compare against a random-permutation null; chance-level matching confirms indices are not aligned.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Methodology states that D_CoT and D_NoCoT are separate SAE dictionaries, then defines hpatch[S] = hCoT[S], hpatch[not S] = hNoCoT[not S]. This presupposes a common feature index set. Since the dictionaries are trained independently on different activation distributions, feature index i in D_CoT and index i in D_NoCoT need not correspond to the same concept. The Top-K ranking by |hCoT - hNoCoT| subtracts vectors from different bases, and decoding the mixed vector requires choosing one of the two decoders, which will read the other component as if it were its own basis. No alignment procedure, permutation, or shared-dictionary training is reported. Therefore the observed log-probability increases in Pythia-2.8B and the absence of effect in 70M cannot be attributed to 'CoT-reasoning features'; the intervention may simply inject arbitrary coordinates into an incompatible dictionary. The random-K 'widely distributed' result is also uninterpretable for the same reason. This is the most load-bearing concern because the causal and scale-threshold claims in the abstract rest entirely on the patching operation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a feature-level causal analysis of whether chain-of-thought (CoT) prompting induces faithful internal reasoning. For Pythia-70M and Pythia-2.8B on GSM8K, the authors train separate sparse autoencoders (SAEs) on CoT and NoCoT residual activations at layer 2, then perform activation patching by replacing selected feature coordinates of the NoCoT sparse code with those of the CoT sparse code. They report that patching a small set of CoT features into NoCoT runs increases answer log-probabilities in the 2.8B model but not in the 70M model, that interpretation scores for SAE features are higher under CoT in the larger model, and that CoT activations are sparser. They interpret these results as evidence of a scale threshold for CoT-induced faithful, distributed internal features. The paper also introduces Top-K and Random-K patch curves and provides code and configuration files.","tokens_in":17987,"tokens_out":4984,"duration_ms":52130,"significance":"If the causal claims were supported, the paper would be a notable step in mechanistic interpretability of CoT reasoning: it would provide feature-level, intervention-based evidence about faithfulness, introduce a patch-curve methodology, and report a scale-dependent effect. The paper is transparent about several limitations, states that code is available, and uses established tools (SAEs, activation patching). However, the central causal claim depends on an unverified assumption that feature indices in two separately trained SAE dictionaries are semantically aligned. Because that assumption is not tested or established, the main experimental result cannot currently be interpreted as measuring CoT features, and the scale-threshold conclusion is not supported by the described procedure. The descriptive observations about activation sparsity remain potentially useful but do not carry the causal conclusion.","major_comments":[{"comment":"The patching operation hpatch[S] = hCoT[S], hpatch[not S] = hNoCoT[not S] presumes that feature index i in DCoT and index i in DNoCoT refer to the same semantic feature. The paper states that two separate SAE models are trained to obtain 'distinct feature dictionaries' and provides no alignment, permutation, or shared-dictionary procedure. Consequently, the Top-K selection based on |hCoT − hNoCoT| subtracts sparse codes written in different bases, and decoding the resulting mixed vector requires choosing one of the two dictionaries, which will misread the other component. The observed log-probability increases in Pythia-2.8B and the absence of effect in Pythia-70M therefore cannot be attributed to CoT-reasoning features; the causal claim and the scale-threshold claim are unsupported as described. A shared dictionary for both conditions, a validated feature-alignment step, or a reformulated intervention is required.","section":"Methodology, Causal Intervention"},{"comment":"The Random-K baseline is described as 'uniformly sampled from the full dictionary.' In a sparse code, most features are zero for a given input, so random sampling from the full dictionary should mostly select inactive features whose values are unchanged by patching, diluting any effect relative to Top-K. The reported result that Random-K outperforms Top-K in Pythia-2.8B is therefore surprising and unexplained. The paper must clarify whether sampling was restricted to active CoT features, and should report the fraction of zero activations in the dictionary, error bars on the patch curves, and confidence intervals for the random baseline.","section":"Experiment Setup and Results (Random-K)"},{"comment":"Multiple statements report p-values for the patching effects (e.g., Figure 5 and Figure 6 captions: 'p < 0.001'; Appendix: 'All patching effects achieved statistical significance (p < 0.001)') without describing the statistical test. It is not stated whether the test is paired across the same set of problems, how the 'up to 1000 problem pairs' are used, or whether any multiple-comparison correction is applied across K values, dictionary ratios, and model sizes. The test procedure, effect sizes, and confidence intervals should be reported.","section":"Results and Appendix (statistical significance)"},{"comment":"The abstract and conclusion claim a 'clear scale threshold' based on two model sizes, Pythia-70M and Pythia-2.8B, evaluated at a single layer (layer 2) and a single token position (final token). A threshold claim requires more than two scale points; the evidence supports at most a scale-dependent effect. The language should be softened, or the claim should be supported with intermediate model sizes and additional layers and token positions.","section":"Abstract and Conclusion (scale threshold)"}],"minor_comments":[{"comment":"The appendix contains an incomplete cross-reference: 'We will further explain this phenomenon through an analysis of feature sparsity structure in Section .' The section number is missing.","section":"Appendix, cross-reference"},{"comment":"The GitHub URL in the paper is given as 'https://github.com/sekirodie1000/cot faithfulness' with a space; this should be URL-encoded or replaced with a working repository link.","section":"Reproducibility link"},{"comment":"There are several typographical errors, including 'reverse-engineere' in Related Work and the repeated phrase 'combining explanation scores with results from explanation scores' in the Semantic Interpretation section.","section":"Related Work and Appendix, typos"},{"comment":"The terms 'explanation score' and 'interpretation score' are used interchangeably; the definitions should be unified, and the near-zero mean scores (e.g., 0.056 vs -0.013 for 2.8B) should be interpreted with appropriate caution rather than described as evidence of 'semantically coherent' features.","section":"Semantic Interpretation, terminology"},{"comment":"The claim that the paper is 'the first to apply SAE-based feature extraction in the context of CoT prompting' is too strong given that Dutta et al. (2024) is cited as examining CoT reasoning mechanistically; the novelty statement should be qualified.","section":"Related Work, novelty claim"}],"recommendation":"major_revision","confidential_remarks":"The central alignment flaw identified in the major comments explains the reader's rejection and is, in my view, the decisive issue: the patching experiment is semantically undefined across separately trained dictionaries. Unlike some other problems, this is in principle fixable by retraining a shared SAE on pooled CoT/NoCoT activations or by providing a validated dictionary-alignment procedure, so I am not recommending outright rejection. However, the revision would require new experiments rather than textual changes, and the scale-threshold and 'first feature-level causal study' claims should be substantially tempered. I would also note the unusual choice to evaluate on the GSM8K training split, which limits generalization claims, and the missing statistical details for the p-values."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a real descriptive finding but the causal headline doesn't survive contact with the method. CoT and NoCoT SAE dictionaries are trained separately and never aligned, so the patching operation swaps arbitrary coordinate indices between two different bases. The stress-test note is right: hpatch[S]=hCoT[S], hpatch[not S]=hNoCoT[not S] assumes feature i means the same thing in both dictionaries, and no permutation, shared dictionary, or alignment check is reported. That makes the log-probability increases in 2.8B and the absence in 70M uninterpretable as evidence about CoT features.\n\nWhat's actually good: the paper ships code, uses two Pythia sizes, and gives a clear workflow. The random-K baseline and patch curves are a sensible way to ask whether useful information is concentrated or distributed, and the descriptive sparsity and interpretability results are worth reporting even without the causal overlay. The finding that CoT leads to sparser activations, especially at 2.8B, is worth taking seriously.\n\nThe soft spots are in proportion: the confounding of CoT with the presence of three few-shot examples is serious, because noCoT gets no examples at all. Patching only layer 2 at the final token is a shallow snapshot; the paper acknowledges this, but it limits the scope of any causal statement. The uniform \"p<0.001\" across many K values and two dictionary ratios looks like multiplicity is not being handled. None of these rescue the causal claim; the alignment problem alone is enough. The citation pattern is fine—they cite Makelov et al. and Wu et al. on interpretability illusions—but they never connect that concern to their own unaligned patch.\n\nWho should read it: interpretability researchers working on CoT faithfulness, and people who want a clean example of why SAE-based intervention needs a common dictionary. I would send it to peer review—not because the current causal claims are acceptable, but because the question is important and a competent referee can force a fix or a reframing. As is, the conclusion should not be cited.","headline":"Descriptive results are worth a look, but the causal claim is built on unaligned SAE dictionaries and a CoT-vs-few-shot confound.","tokens_in":18428,"tokens_out":3597,"would_cite":false,"duration_ms":38006,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that chain-of-thought prompting changes a large language model's internal features into sparser, more interpretable, causally effective units—but only past a capacity threshold, and it tests this by swapping features…","keywords":["chain-of-thought prompting","mechanistic interpretability","sparse autoencoders","activation patching","faithfulness","language model reasoning","scale threshold","GSM8K"],"falsifier":"Train a single SAE on pooled CoT and noCoT activations so the dictionary is shared, then repeat the patching; if the log-probability gains disappear, the original effect depends on coincidental index alignment rather than on CoT features. A complementary check is token-level path patching: if the final-token patch still helps when all earlier CoT tokens are masked, the effect is not a trace of the reasoning process itself.","tokens_in":17551,"feed_emoji":"🧠","tokens_out":10948,"duration_ms":98396,"temperature":0.7,"pith_summary":"The paper asks whether chain-of-thought prompting changes how a language model actually reasons, not just what it outputs. To test this, it trains sparse autoencoders on the hidden activations of a 70-million-parameter and a 2.8-billion-parameter model solving grade-school math problems under chain-of-thought and plain prompting, then patches the extracted features between the two conditions. The central finding is that swapping chain-of-thought features into a plain run substantially raises the log-probability of the correct answer in the larger model but has no reliable effect in the smaller one, a scale threshold. Chain-of-thought also makes activations sparser and its features more interpretable in the larger model. The paper reads this as evidence that chain-of-thought can induce genuinely faithful internal structure when the model has enough capacity.","feed_headline":"Chain-of-thought creates real reasoning features in large models only","feed_subtitle":"Patching chain-of-thought features into a plain run lifts correct-answer confidence — but only at 2.8B scale.","key_machinery":"The work is carried by a three-part mechanism: sparse autoencoders (SAEs) that compress the residual-stream activation at the final token into a sparse set of interpretable feature directions; activation patching that constructs a hybrid feature vector $h_{patch}[S] = h_{CoT}[S]$, $h_{patch}[\\bar{S}] = h_{NoCoT}[\\bar{S}]$, decodes it back into activation space, and measures the change in log-probability of the correct answer; and patch-curve analysis that varies K for top-K and random-K feature subsets to see whether causal signal is concentrated or distributed. The sparsity analysis adds a structural correlate: under CoT, most neurons are near zero and each SAE feature engages few neurons, with high variance across features in the larger model, which the paper calls structured sparsity.","core_discovery":"On the paper's own terms, the discovery is a scale-dependent signature of chain-of-thought faithfulness. In the 2.8-billion-parameter model, replacing a selected set of feature values in a noCoT forward pass with values taken from a CoT forward pass raises the answer log-probability from about 1.2 to 4.3, with the top-K patch curves jumping by more than 3.2 log-prob at K=2 under a sparser dictionary; the reverse direction has little effect. In the 70-million-parameter model, the same intervention produces unstable, often negative log-probability changes. A second finding is that random-K patching of CoT features often beats top-K patching in the larger model, indicating that CoT's causal information is spread across many moderately activated features rather than concentrated in the most activated directions. The paper also reports that CoT prompts yield significantly sparser residual activations and higher feature explanation scores in the larger model. Together, these results support the conclusion that CoT reshapes internal computation into sparser, more modular, causally effective features—but only above a capacity threshold.","pith_inferences":["If the scale threshold generalizes beyond this model family, a practical consequence is that CoT faithfulness can be screened cheaply by measuring activation sparsity or random-patch transfer before doing full mechanistic mapping.","The random-K-over-top-K result suggests a testable hypothesis for feature steering generally: ranking features by activation or by $|h_{CoT} - h_{NoCoT}|$ may select outliers, while a diverse sample of the feature vector is a better causal proxy; this could be checked on other tasks and models.","The alignment caveat is where I would place my own concern: because the CoT and noCoT SAE dictionaries are trained separately, the patching result implicitly assumes the same index in the two dictionaries corresponds to the same semantic feature; an alignment-free formulation of the test would harden the causal conclusion.","A sharper estimate of the threshold would come from repeating the patch-curve comparison at intermediate model sizes (around 1B parameters) and on multi-step non-math reasoning, which the paper does not do."],"forward_implications":["Above the observed capacity threshold, CoT-elicited features act as transferable causal units: inserting a small set of them into a noCoT run shifts the model's output toward the correct answer, and the direction is asymmetric—CoT-to-noCoT helps while noCoT-to-CoT does not.","Because random feature subsets outperform top-K subsets in the larger model, selection by activation magnitude is not a reliable way to find causally important CoT features; a distributed, coverage-based selection policy should be considered.","CoT's benefits are not purely superficial: in large models it measurably increases residual-stream sparsity and feature interpretability, so CoT can be described as a structured prompting method rather than only a surface scaffold.","The 70M result acts as a lower bound for the approach: in low-capacity models, CoT features do not transfer causally and can even degrade performance, so future feature-level studies of CoT should expect and report a scale threshold rather than assuming uniform effects."],"supporting_citations":[{"why":"Introduces chain-of-thought prompting, the phenomenon whose internal faithfulness is being tested.","marker":"Wei et al. 2022"},{"why":"Provides the GSM8K grade-school math benchmark used for all reasoning evaluations.","marker":"Cobbe et al. 2021"},{"why":"Supplies the sparse-autoencoder method for learning interpretable, monosemantic features from activations.","marker":"Cunningham et al. 2023"},{"why":"Establishes the dictionary-learning/sparsity objective and monosemanticity framing that justify feature-level analysis.","marker":"Bricken et al. 2023"},{"why":"Introduces activation patching, the intervention tool used to swap features between CoT and noCoT runs.","marker":"Meng et al. 2022"},{"why":"Provides the LLM-based explanation and scoring method used to measure feature interpretability.","marker":"Bills et al. 2023"},{"why":"Shows that small sets of SAE features form sparse causal circuits, the precedent for expecting a few features to carry causal weight.","marker":"Marks et al. 2024"},{"why":"Poses the interpretability-illusion caveat that feature-level patching must overcome to be causally meaningful.","marker":"Makelov, Lange, and Nanda 2023"}],"fun_headline_variants":["CoT's causal features only aid reasoning in 2.8B models","Patching CoT features lifts answer confidence 3.5x — at 2.8B only","Random CoT features beat top-K: reasoning is distributed","Scale threshold: CoT's internal reasoning features emerge at 2.8B","CoT makes 2.8B models sparse and patchable — not 70M"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that feature index i in the CoT sparse-autoencoder dictionary means the same thing as feature index i in the separately trained noCoT dictionary; if the two dictionaries are not aligned, the patch does not swap CoT features but writes arbitrary coordinates into the noCoT representation.","fun_headline_variants_meta":{"raw":{"variants":["CoT's causal features only aid reasoning in 2.8B models","Patching CoT features lifts answer confidence 3.5x — at 2.8B only","Random CoT features beat top-K: reasoning is distributed","Scale threshold: CoT's internal reasoning features emerge at 2.8B","CoT makes 2.8B models sparse and patchable — not 70M"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000559,"raw_usage":{"total_tokens":2698,"prompt_tokens":1026,"completion_tokens":1672,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":1565}},"tokens_in":642,"tokens_out":1672,"duration_ms":13405,"temperature":1.0,"reasoning_tokens":1565,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:16:00.282945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a single SAE on pooled CoT and noCoT activations so the dictionary is shared, then repeat the patching; if the log-probability gains disappear, the original effect depends on coincidental index alignment rather than on CoT features. A complementary check is token-level path patching: if the final-token patch still helps when all earlier CoT tokens are masked, the effect is not a trace of the reasoning process itself.","supporting_citations":[],"review_version":1}