{"id":"acceaf22-8f48-4a01-8e2e-1cce67f9edb3","arxiv_id":"2412.03944","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Chain-of-thought prompting makes LLMs imitate exemplar formatting, concentrate their final answer probabilities, and activate a wider set of final-layer neurons.","lead":"This paper studies how chain-of-thought prompting changes what happens inside large language models as they answer reasoning questions. It reports that models copy the format of the examples while still using their own knowledge, and that chain-of-thought makes their final answer predictions more confident while switching on more neurons in the last layers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.5 activation-range result is confounded by output length: CoT generations are far longer than standard answers, so a larger aggregated active-neuron set may be a length artifact rather than evidence of broader knowledge retrieval.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the activation comparison in Section 3.5 is not length-controlled, so the reported broader activation range under CoT could be an artifact of long CoT outputs. My stress-test confirms that this is the most serious threat to the central claim. The paper's three findings are otherwise plausible and internally consistent, but the activation claim is the one that carries the strongest mechanistic interpretation ('deeper and broader knowledge retrieval') and it is the least protected against a trivial alternative explanation. The 'imitation' claim is partly definitional, but the authors do include a transfer test and a correlation with accuracy, which gives it some evidential support; the logits-entropy claim is also suggestive, though lacking statistical tests. Neither is as severely threatened as the activation claim, where the length confound alone can explain the headline pattern. Because the concern is concrete and testable (per-token normalization or a length-padded control would settle it), the appropriate action is not to reject the paper but to require the authors to provide this analysis, consistent with the reader's CONDITIONAL verdict. I therefore recommend no change to the reader's verdict.","tokens_in":31133,"tokens_out":3162,"duration_ms":32141,"concrete_test":"Recompute the Section 3.5 activation statistics on Gemma2-9b for GSM8K using per-token normalization: for each generated token, compute the activation range per FFN layer, then average over tokens within each response, and compare Standard vs CoT. As a second control, generate Standard responses padded to the same length as CoT (e.g., continuing with neutral filler after the answer) and rerun the original pooled analysis. If per-token ranges are similar, or if padding reproduces the wider range, the 'broader activation' conclusion is a length artifact; if CoT per-token range remains larger and padded Standard remains small, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central activation claim in Section 3.5 ('CoT activates a wider range of neurons... suggesting more extensive knowledge retrieval') rests on comparing activation range and intensity between Standard and CoT prompts. The paper reports these metrics as 'the proportion of activated neurons relative to the total number of neurons in the FFN layer' and 'the averaged activation value of the activated neurons,' but the text does not specify how the aggregation over generated tokens is performed. Under the most natural reading, the metrics are computed by pooling all activations across the entire generated sequence. CoT generations routinely contain many more tokens than Standard responses: Standard answers are often just 'The answer is 6.', while CoT outputs include multi-step reasoning of 50–200 tokens. Even if individual tokens activate disjoint sets of neurons, the pooled active-neuron set (or the range of activation values) will mechanically grow with sequence length. Thus the observed 'broader activation range' under CoT may simply reflect that more tokens were generated, not that CoT elicits deeper or broader knowledge retrieval. The paper's Limitations section acknowledges that only FFN layers were examined, but it does not address this sequence-length confound. This is the load-bearing weakness because the headline mechanistic interpretation in the abstract and conclusion—that CoT enables 'more extensive knowledge retrieval'—is directly supported by Section 3.5, and no control condition or per-token normalization is provided to rule out the trivial length effect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of how Chain-of-Thought (CoT) prompting changes the behavior of large language models. Using 50 randomly sampled questions from each of nine reasoning datasets and four models (Gemma2 2B/9B/27B and LLaMA2-13B), the authors study three aspects: (1) whether models imitate CoT exemplar formats, measured through predefined test-point word lists and a cross-task transfer test; (2) how token-level logits and the final answer distribution change, measured through token probability curves, kernel density estimates of the phrase \"the answer is\", and entropy of top-k answer probabilities; and (3) how FFN neuron activation changes, measured through activation range and intensity across the final 20 layers. The paper concludes that LLMs imitate exemplar formats while also integrating question understanding, that CoT produces a more concentrated final logits distribution, and that CoT activates a broader set of final-layer neurons, interpreted as more extensive knowledge retrieval. The paper closes with a Limitations section acknowledging limited task scope, fixed prompt types, FFN-only analysis, and the exploratory nature of the study.","tokens_in":31406,"tokens_out":4288,"duration_ms":42500,"significance":"If the findings were rigorously established, the paper would provide useful observational evidence about the internal and external effects of CoT prompting, complementing existing hypotheses from Saparov and He, Madaan et al., Geva et al., and Rai and Yao. The study has real strengths: it spans multiple model families and sizes, uses publicly available models, provides the full prompts in the appendix, and attempts to relate imitation, logits, and neuron activation to each other. However, the current evidence is largely descriptive and lacks the controls and statistical machinery needed to support the mechanistic claims. The most important strength is the breadth of datasets and models; the main weakness is that the headline conclusions depend on comparisons that are confounded by output length, on a partly circular test-point metric, and on visual inspection of scatter and density plots without confidence intervals or significance tests. As it stands, the paper is a useful exploratory report rather than a definitive mechanistic account.","major_comments":[{"comment":"The activation-range comparison is confounded by sequence length. The text defines Activation Range as \"the proportion of activated neurons relative to the total number of neurons in the FFN layer\" and Activation Intensity as \"the averaged activation value of the activated neurons,\" but it does not state whether these metrics are computed per generated token and then averaged, or pooled over all tokens in a generation. CoT outputs are routinely much longer than Standard outputs; under a pooled reading, the union of active neurons will mechanically grow with output length even if per-token activation is unchanged. A length-matched control (e.g., per-token averaged activation, or comparing only the first N tokens) is necessary before interpreting the observed broader activation range as \"more extensive knowledge retrieval.\" The Limitations section does not mention this confound.","section":"Section 3.5, Figure 7"},{"comment":"The test-point measure of imitation is partly circular. The lists of \"time,\" \"action,\" and \"loc&peo\" words are manually derived from the exemplars (Table 18), and the \"number\" category is extracted from the questions; observing these words in CoT-generated text is therefore expected under the very definition of the metric. The imitation claim needs a control condition, such as a Standard-prompt baseline or a matched set of control words that are frequent in English but absent from the exemplars, together with effect sizes for the match proportions. Without such a baseline, the takeaway that LLMs \"imitate exemplar formats\" is not strongly supported.","section":"Section 3.3 and Appendix B"},{"comment":"All quantitative comparisons rest on only 50 examples per dataset, and no confidence intervals, error bars, or statistical tests are reported. For example, Figure 6 shows overlapping entropy scatter points between CoT and Standard conditions, and Figure 7 compares activation range and intensity with summary boxes only. The authors should add bootstrap confidence intervals, paired comparisons on the same questions, or per-dataset effect sizes so that the reader can assess whether the observed differences are larger than sampling noise.","section":"Section 3.1 and Figures 5-7"},{"comment":"The claim that \"the final output shows a more concentrated logits distribution\" relies on entropy values computed after selecting the top k probabilities and normalizing them, where k is the answer-space size. Because the top-k selection is informed by the observed outputs, and because the scatter plots show substantial overlap between conditions, the analysis should state the null hypothesis explicitly and report a quantitative comparison (e.g., paired differences between CoT and Standard for the same questions). The current presentation does not rule out the possibility that the difference is driven by a small number of outlier examples.","section":"Section 3.4 and Appendix H"}],"minor_comments":[{"comment":"There is a typo: \"avialable\" should be \"available,\" and \"Chainof-Thought\" should be \"Chain-of-Thought.\"","section":"Abstract"},{"comment":"The entry for \"number\" test points reads \"it was obtained by regular expression,\" which appears to be a placeholder; please replace it with the actual list or the precise extraction rule.","section":"Appendix B, Table 18"},{"comment":"The header says \"Top 10 tokens,\" but the table rows include subword-like items such as \"pped\" and \"aten\"; please clarify whether these are subword tokens and how spaces and punctuation were handled.","section":"Table 1"},{"comment":"The imitation heuristic uses an ad hoc threshold of more than four occurrences of verbs like \"flips, is, was, are, be, were\" to decide whether the model generated new entities; the choice of threshold and the list of verbs should be justified or varied in a sensitivity analysis.","section":"Appendix D"},{"comment":"Figures 15-17 show probability traces for a single example per dataset per model; the paper should state how representative these examples are, since they are used to support the qualitative claim that CoT causes \"sharp drops and oscillations\" in token probabilities.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The manuscript is an exploratory observational study with several plausible and potentially interesting findings, but the current evidence is not yet sufficient for the mechanistic claims made in the abstract and conclusion. The activation analysis is confounded by output length, the imitation metric has a circularity problem, and the quantitative comparisons lack statistical grounding. These are fixable within the scope of the manuscript: the authors could add length-matched controls, a non-circular baseline, and bootstrap confidence intervals or paired tests. I do not see a fatal flaw in the overall direction, but the load-bearing claims need substantial strengthening. I would also encourage the authors to release their code and data during revision rather than only upon acceptance, since the paper's value depends heavily on the details of the metrics and thresholds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an honest, clearly written exploratory paper that triangulates three existing CoT hypotheses—format imitation, logits concentration, and broadened FFN activation—in one battery across four models and nine datasets. It is not a breakthrough, but it is a useful consolidation. The entropy finding is the strongest piece: at the answer-generation step, the top-k probabilities line up with the k answer options, and CoT shifts mass onto the correct option. The transfer-test design for imitation is also a decent idea, even if the metric is partly self-fulfilling.\n\nThe soft spots are real but not fatal. The activation-range result in Section 3.5 has a length confound: CoT generations are much longer than standard answers, and the paper never says how activation metrics are aggregated over tokens. If they pool across the generated sequence, a larger active-neuron set under CoT is exactly what you'd expect from more tokens, not from deeper retrieval. The authors need a per-token or length-matched baseline before the 'broader knowledge retrieval' wording is justified.\n\nThe other weaknesses are straightforward: 50 examples per dataset, no error bars or significance tests, and the code/data are promised only upon acceptance. The imitation metric also inherits the authors' test-point word lists, so some of the imitation signal is baked into the measurement. The limitations section is candid about prompt and dataset coverage but does not mention the length confound or the absence of statistics.\n\nNet: worth a serious referee, but my own verdict would be conditional. I'd ask for length-controlled activation analysis, statistical summaries, and the artifacts before believing the mechanism claims. The entropy part and the transfer matrix could survive on their own. I would not cite it yet, but I'd bring it to a reading group to discuss the measurement methodology.","headline":"A readable but statistically thin empirical battery; the activation claim is likely confounded by output length.","tokens_in":31940,"tokens_out":1725,"would_cite":false,"duration_ms":17684,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that chain-of-thought prompting produces three measurable internal changes in large language models: imitation of exemplar format, concentration of final-answer logits, and broader final-layer neuron activation.","keywords":["chain-of-thought","large language models","logits","entropy","neuron activation","feed-forward layers","prompt imitation","reasoning tasks"],"falsifier":"Run a length-matched control: take standard-prompt answers, pad them to chain-of-thought length with neutral filler reasoning, and recompute final-layer activation range and intensity; if padded standard outputs show the same broad activation, the knowledge-retrieval interpretation fails.","tokens_in":30932,"feed_emoji":"🧠","tokens_out":6763,"duration_ms":55604,"temperature":0.7,"pith_summary":"The paper tries to establish that chain-of-thought prompting changes a language model in three observable ways: it imitates the exemplar format while drawing on its own understanding of the question, it produces a more concentrated probability distribution over the final answer, and it activates a wider set of neurons in the final feed-forward layers. These observations are offered as evidence about how chain-of-thought works internally, not just that it improves accuracy. The authors measure imitation with four word categories, logits with probability traces and entropy over candidate answers, and activation with range and intensity in feed-forward layers across four model sizes.","feed_headline":"Chain-of-thought narrows answer logits and widens neuron firing","feed_subtitle":"Across four models, CoT prompts tighten the final answer distribution while activating a broader neuron set.","key_machinery":"The analysis rests on three measurement tools. Test points are four predefined word categories (time, action, location/person, number) matched by exact string against generated text to quantify exemplar imitation and transfer. Logits are normalized to probabilities and tracked horizontally across generated tokens and vertically as the entropy of the probability distribution over the answer space. Neuron activation is measured in the feed-forward layers by two metrics: activation range, the proportion of neurons with value greater than zero, and activation intensity, the average value of active neurons. These tools make the chain-of-thought versus standard difference legible at the decoding, projection, and activation levels.","core_discovery":"The paper's central claim is that chain-of-thought prompting alters a model's internal and external behavior in three discernible ways. First, the model does not simply copy the exemplars: it adopts the sequential format markers (like 'first', 'so', 'then', arithmetic operators) while supplying content driven by its own understanding of the question, and this imitation-plus-understanding is what yields correct answers. Second, during generation the token-level logits fluctuate more than under standard prompting, yet at the final answer the probability distribution over candidate answers is more concentrated, as measured by lower entropy on tasks with finite answer spaces. Third, across the final twenty feed-forward layers, chain-of-thought expands the activation range (fraction of neurons with positive values) while lowering activation intensity, which the authors interpret as broader retrieval of pre-trained knowledge.","pith_inferences":["The paper does not control for output length in the activation comparison; a length-matched control would test whether the broader neuron range reflects wider knowledge retrieval or simply more generated tokens.","The imitation metric relies on predefined word lists, so the imitation finding is partly shaped by the chosen test points; an open-vocabulary measure of format adherence could confirm the claim.","If concentrated answer logits track correctness, chain-of-thought could double as a calibration signal; the paper's correct/wrong scatter plots suggest such a relationship but stop short of a calibration analysis.","The final-layer activation expansion could be probed by ablating the newly activated neurons; if answers degrade, the broadened retrieval is causally implicated, not merely correlated."],"forward_implications":["Chain-of-thought's benefit is not pure imitation: transfer tests show the model follows the exemplar format while using its own knowledge, and better imitation tracks better accuracy.","Logits behavior splits into two phases: turbulent intermediate reasoning, then a sharpened final answer distribution, suggesting chain-of-thought acts as a reasoning scratchpad that converges at the end.","The activation signature of broader range and lower intensity in final layers provides a quantitative marker for deeper knowledge retrieval that later work can use to compare prompt designs.","Entropy over the answer space offers a cheap confidence signal: chain-of-thought's more concentrated distribution at the answer token could be used to flag model certainty on finite-answer tasks.","The patterns reproduce across all four tested model sizes and across arithmetic, commonsense, and symbolic reasoning tasks."],"supporting_citations":[{"why":"Supplies the chain-of-thought prompting method and the exemplars that this paper modifies; the central phenomenon under study.","marker":"(Wei et al., 2022)"},{"why":"Empirical study of what matters in chain-of-thought prompts; motivates the imitation-versus-understanding question and supplies the Bamboogle prompts.","marker":"(Wang et al., 2023)"},{"why":"The hypothesis that models mimic exemplar formats; directly tested by the test-point and transfer experiments.","marker":"(Saparov and He, 2023)"},{"why":"Identifies transformer feed-forward layers as key-value memories; justifies measuring activation in FFN layers.","marker":"(Geva et al., 2021b)"},{"why":"Prior neuron-activation analysis of chain-of-thought; the activation-range and intensity metrics follow this line.","marker":"(Rai and Yao, 2024)"},{"why":"Provides the GSM8K dataset used as an arithmetic benchmark.","marker":"(Cobbe et al., 2021)"},{"why":"Provides the SVAMP dataset used as an arithmetic benchmark.","marker":"(Patel et al., 2021)"},{"why":"Kernel density estimation used to compare chain-of-thought versus standard logits distributions.","marker":"(Parzen, 1962)"},{"why":"Entropy as the measure of distribution concentration over answer candidates.","marker":"(Pereira et al., 1993)"}],"fun_headline_variants":["CoT prompts tighten answer logits and broaden neuron use","Chain-of-thought: imitate example style, think own answer","CoT reasoning: more logit fluctuation, sharper final answers","CoT alters neuron firing and concentrates final probabilities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the wider neuron activation seen under chain-of-thought is not a side effect of chain-of-thought generating far longer outputs, since activation range and intensity are aggregated over generated tokens without a length-controlled baseline; the paper's own limitations section also restricts generalization to the tested datasets and prompt conditions.","fun_headline_variants_meta":{"raw":{"variants":["CoT prompts tighten answer logits and broaden neuron use","Chain-of-thought: imitate example style, think own answer","CoT reasoning: more logit fluctuation, sharper final answers","CoT alters neuron firing and concentrates final probabilities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3601,"prompt_tokens":833,"completion_tokens":2768,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":2701}},"tokens_in":449,"tokens_out":2768,"duration_ms":18201,"temperature":1.0,"reasoning_tokens":2701,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:55:35.055693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a length-matched control: take standard-prompt answers, pad them to chain-of-thought length with neutral filler reasoning, and recompute final-layer activation range and intensity; if padded standard outputs show the same broad activation, the knowledge-retrieval interpretation fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Kernel density estimation used to compare chain-of-thought versus standard logits distributions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Entropy as the measure of distribution concentration over answer candidates."}],"review_version":1}