{"id":"78a8e22c-a507-4189-b1df-bdcc775cf3e2","arxiv_id":"2411.11371","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Thinking Tokens underperform Chain-of-Thought on arithmetic and QA benchmarks, and the paper attributes this to noisy gradients from a single shared token embedding.","lead":"This paper tests whether Thinking Tokens, a special placeholder token that lets a language model compute internally, actually help with reasoning. It finds they usually underperform Chain-of-Thought and argues the cause is noisy gradients from a single shared token embedding.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The root-cause claim is not tested: two distinct thinking-token embeddings move more, but the paper never shows they improve accuracy. Without a two-token performance comparison, noisy gradients remain an untested hypothesis.","rationale":"The reader's weakest assumption is that small embedding movement and low cumulative gradient (Figures 2 and 3, Section 4.2) measure noisy gradients, and that this noise causes underperformance, rather than the model simply ignoring a token it has no pressure to use. This is exactly the load-bearing gap I identify. The paper's mechanistic claim depends on showing that reducing the supposed noise source (by using two distinct embeddings) improves task accuracy. Figure 4 and Figure 5 show only that embeddings move more; they do not show better accuracy. Without that comparison, the observed gradient statistics are consistent with multiple explanations, including the trivial one that a model does not update an embedding it has learned to ignore. The empirical comparison in Table 1 is useful evidence for the negative result that a single Thinking Token underperforms Chain-of-Thought, but the headline abstraction and Section 5.1 go beyond it. The theoretical section is not a derivation that would independently support the causal claim; Equations (2)-(4) simply restate the reuse structure and label the consequence 'noisy'. The concrete test I propose directly settles whether the two-token intervention improves performance. If it does not, the root-cause claim should be removed or substantially weakened, and the paper should be reframed as an empirical negative result about single Thinking Tokens. If it does, the mechanism is supported. Either way, the paper's current conditional status is appropriate: the empirical comparison is worth publishing, but the root-cause claim needs an additional performance experiment before it can stand. My read therefore does not change the reader's verdict.","tokens_in":4669,"tokens_out":2819,"duration_ms":30560,"concrete_test":"Train the two-token variant with distinct embeddings ('t' and 'ts') on the same digit-multiplication and GSM8k setups as Section 4, and evaluate it alongside single-TT, CoT, and TT+CoT. Report exact-match accuracy with multiple seeds and error bars. If the two-token variant does not outperform single-TT, or does not approach CoT accuracy, then Figures 4-5 are epiphenomenal and the noisy-gradient root-cause claim is unsupported. If the two-token variant does improve accuracy substantially, the mechanism gains direct support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is the claim that Thinking Tokens underperform because a single shared embedding produces inconsistent learning signals and noisy gradients (Sections 3, 5.1). The empirical evidence for this mechanism is Figures 2-5: one embedding barely moves and has low cumulative gradient, while two embeddings move more and receive larger cumulative gradients. This evidence is correlational and misses the causal link to task performance. A token embedding can remain near initialization simply because the model learns to ignore a token it is not pressured to use, not because gradients are noisy. Likewise, two embeddings moving more shows only that the optimizer updates them more; it does not show that the model reasons better. Table 1 compares single-TT, CoT, and TT+CoT, but never reports a two-token condition. Thus the paper never tests whether reducing gradient noise by using two distinct embeddings actually improves accuracy. The theoretical section does not close this gap: Equations (2)-(4) assert that gradient updates are 'noisier' or 'inconsistent' when an embedding is reused, but no derivation shows that this noise is the cause of the observed underperformance rather than a symptom of the model ignoring an optional token. If the two-token variant also fails to improve accuracy, the entire mechanistic explanation collapses, even though the empirical finding that single-TT underperforms CoT may remain true.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares Thinking Tokens (TT), Chain-of-Thought (CoT), TT+CoT, and a baseline on digit multiplication and on GSM8k and OpenBookQA, and reports that TT underperforms CoT. It hypothesizes that this underperformance is caused by the reuse of a single TT embedding, which produces inconsistent learning signals and noisy gradients. To test this, the authors measure embedding movement and cumulative gradient magnitude for one- and two-token variants, finding that two distinct TT embeddings move more and receive larger cumulative gradients. The paper concludes that TT's reliance on a single embedding is the root cause of its poor performance.","tokens_in":4916,"tokens_out":3260,"duration_ms":32443,"significance":"If the empirical comparison is reliable, the paper provides a useful negative result: an unsupervised 'thinking token' mechanism does not match explicit step-by-step supervision, and in some settings it can degrade performance. The gradient/embedding measurements are an appropriate way to probe what the model learns about a newly introduced token, and the two-token condition is a falsifiable intervention. However, the central causal claim—that noisy gradients from a single shared embedding explain the underperformance—is not tested end-to-end, because no accuracy result is reported for the two-token variant. The current significance is therefore mainly empirical (the negative result), not mechanistic.","major_comments":[{"comment":"The abstract states that Thinking Tokens 'marginally improves performance,' but Table 1 shows that TT reduces GSM8k exact match relative to baseline (4.51 vs 6.30) and produces no change on OpenBookQA (37.2 vs 37.2). The abstract therefore contradicts the paper's own main table and should be corrected.","section":"Abstract and Table 1"},{"comment":"The causal claim that a single embedding causes underperformance through noisy gradients is not tested. The two-token variant shows larger embedding movement and larger cumulative gradients, but no accuracy numbers are reported for the two-token condition. Without showing that using two distinct TT embeddings improves task performance (or at least does not degrade it), the evidence does not establish that the single embedding is the cause of the underperformance; the improved gradient statistics could simply reflect that the optimizer updates a new token more, without affecting reasoning quality.","section":"§5.1, Figures 4–5"},{"comment":"The theoretical basis for 'noisy gradients' is asserted rather than derived. Equation (4) merely sums per-step gradient contributions; it does not show that the per-step signals have zero mean or that role ambiguity translates into gradient noise. Moreover, the operational definition of noise in §4.2—a low cumulative gradient—is partly definitional: a token that the model largely ignores would also have a small cumulative gradient. The paper should either provide a direct measure of gradient cancellation (e.g., per-step cosine similarity or variance) or temper the claim that low cumulative gradient is evidence of noise.","section":"§3, Eq. (4) and §4.2"},{"comment":"The main text says the digit-multiplication model is trained for 100 epochs, while the appendix says 50 epochs. This inconsistency affects reproducibility. Furthermore, no standard deviations, confidence intervals, or multiple-seed results are reported, which is particularly problematic for the small-gap comparisons (e.g., GSM8k TT vs baseline and OpenBookQA TT vs baseline) that the abstract describes as 'marginal.'","section":"§4.1.1 and Appendix A.1.1"}],"minor_comments":[{"comment":"There are grammatical errors in the first sentence, including 'TTs marginally improves performance,' which should be 'marginally improve.'","section":"Abstract"},{"comment":"The x-axis labels in all four figures appear garbled ('x = ( ) + b e p u t' and 'x = ( ) + b e p u t ts'); the figures should be re-generated with readable token labels.","section":"Figures 2–5"},{"comment":"The entry '> 0' for 4-digit multiplication is unsatisfying; the paper should report the actual accuracy or explain why the run was terminated and how that affects the comparison.","section":"Table 1"},{"comment":"The claim that TT+CoT 'perform[s] as if they weren't present' is inaccurate for digit multiplication, where TT+CoT slightly exceeds CoT (92.3 vs 91.9 for 2-digit, 67.8 vs 66.3 for 3-digit); this should be acknowledged.","section":"§6 Discussion"},{"comment":"The text says 'cite as lrfinder' for the learning-rate finder; this placeholder should be replaced with a proper citation or description.","section":"Appendix A.1.2"},{"comment":"The two-token experiment is described only briefly; details about how the two-token model is trained, how many tokens are inserted, and whether it is evaluated on the same benchmarks should be provided.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The negative result on Thinking Tokens is potentially publishable as an empirical observation, but the paper's title and central hypothesis promise a root-cause explanation that the current evidence does not support. The authors should be encouraged to add a two-token accuracy comparison; if that experiment does not change the claims, the paper should be restructured as an empirical study with a clearly labeled hypothesis rather than a validated mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful negative result wrapped in a root-cause claim that the experiments don't actually support. Table 1 clearly shows single Thinking Tokens underperform CoT and can hurt (GSM8k 4.51 vs baseline 6.30; OpenBookQA unchanged). That part is solid and worth knowing. The gradient analysis—one embedding barely moves, cumulative gradient small, two distinct embeddings move more—is a genuinely new diagnostic, and the two-token comparison is a reasonable falsifiable check. But the causal story is never tested. The paper never reports accuracy for the two-token variant. The fact that two embeddings receive larger gradients shows only that the optimizer updates them more, not that the model reasons better. The 'noisy gradients' conclusion is partly definitional: low cumulative gradient is used as evidence of noise, but the alternative explanation—the model learns to ignore an optional token it has no pressure to use—fits the same observations. The theoretical section is schematic; equations (2)-(4) assert inconsistency rather than deriving it. The abstract overstates the results: 'marginally improves' contradicts the GSM8k and OpenBookQA numbers. Also, no error bars or seeds are reported, which matters for small finetuning runs. That said, the paper is not incoherent. It does a controlled comparison and reports honest numbers. The two-token embedding movement is a real observation that could be turned into a proper test if the authors added a two-token condition to Table 1. I'd send this to peer review, but with the expectation that the root-cause claim be either scoped down to 'consistent with noisy gradients' or supported by a performance experiment.","headline":"Useful negative result on Thinking Tokens, but the noisy-gradient root cause is asserted rather than tested because no two-token accuracy is reported.","tokens_in":5451,"tokens_out":1560,"would_cite":false,"duration_ms":15784,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Thinking Tokens, which insert a learnable token to give a model extra computation before answering, only marginally improve reasoning and consistently underperform Chain-of-Thought prompting; the paper argues the cause is the single…","keywords":["Thinking Tokens","Chain-of-Thought prompting","reasoning in language models","gradient analysis","token embeddings","noisy gradients","unsupervised reasoning","arithmetic reasoning"],"falsifier":"Train a single-thinking-token model with an auxiliary objective that forces the token's embedding to receive large, consistent gradients (for instance, by making the token predict the next reasoning step) and check whether accuracy on GSM8k rises to CoT levels; if accuracy stays flat while the embedding moves, the noisy-gradient story is wrong.","tokens_in":4427,"feed_emoji":"🧠","tokens_out":4522,"duration_ms":39118,"temperature":0.7,"pith_summary":"The paper sets out to explain why Thinking Tokens, an unsupervised alternative to Chain-of-Thought prompting, underperform in practice. Its central claim is that the underperformance is not a failure of latent reasoning as an idea but of the mechanism: a single shared token embedding is reused across many contexts, so it receives inconsistent learning signals and noisy gradients. The authors support this with controlled comparisons on arithmetic and natural-language reasoning benchmarks and with gradient analyses showing that one thinking-token embedding barely moves during training, while two distinct token embeddings receive clear gradients and move substantially. If the claim is right, the path forward is richer, more flexible latent vectors rather than abandoning unsupervised reasoning tokens.","feed_headline":"Thinking tokens underperform chain-of-thought in every test","feed_subtitle":"A single shared embedding yields noisy gradients; two token embeddings recover clear learning signals.","key_machinery":"The load-bearing object is the thinking-token embedding $e_{\\mathrm{TT}}$, a single shared vector inserted as an intermediate token in the sequence. The argument is formalized as $\\Delta e_{\\mathrm{TT}} = \\sum_{i} \\nabla L(e^i_{\\mathrm{TT}})$: the update to this one vector is the sum of gradient contributions from every reasoning step in which it appears, and because those steps demand different roles, the sum is noisy and the cumulative gradient is small. The contrast case is CoT, where each step has a distinct token $e^m_{\\mathrm{CoT}}$ with its own isolated gradient $\\nabla L(e^m_{\\mathrm{CoT}})$. The mechanism is tested by comparing embedding movement and cumulative-gradient magnitude for one versus two thinking tokens.","core_discovery":"On the paper's own terms, the discovery is that Thinking Tokens marginally improve accuracy over a no-reasoning baseline and consistently underperform Chain-of-Thought reasoning on digit multiplication, GSM8k, and OpenBookQA, and that this gap is caused by the single-token embedding. Because the same embedding vector is reused at every internal reasoning step, the learning signal it receives averages over conflicting roles, producing high-variance, low-magnitude gradient updates; the embedding therefore stays near its initialization. Introducing two distinct thinking-token embeddings changes the gradient pattern visibly, which the paper takes as empirical validation of the hypothesis.","pith_inferences":["Beyond the paper, the same failure mode likely applies to any dummy token inserted without an explicit training signal: if nothing forces the model to rely on the token, low gradient flow may simply mean the model learned to ignore it, and the causal role of 'noise' remains open.","A testable extension of the paper's logic is that a continuous 'thinking vector' or per-step embeddings, rather than one shared discrete token, should close much of the gap with CoT on arithmetic benchmarks; this follows from the paper's claim but is not itself tested here.","The two-token result suggests that the benefit comes from giving gradient descent distinct coordinates to push on, not from more thinking time; an experiment that increases the number of thinking steps with a single embedding and observes no accuracy gain would sharpen this conclusion.","If the noisy-gradient explanation is correct, then enlarging the embedding dimension or adding an auxiliary prediction loss on the thinking token should increase cumulative gradient and, in turn, accuracy; that is a direct, falsifiable prediction the paper does not run."],"forward_implications":["Thinking Tokens as currently implemented should not be expected to replace Chain-of-Thought prompting on tasks that require structured, multi-step reasoning.","Adding Thinking Tokens to a CoT pipeline contributes little: the TT+CoT results track CoT alone, so the extra token is effectively inert.","The single-embedding design, not latent reasoning in general, is the identified bottleneck; methods that give the model a richer internal representation should be the target of future work.","Embedding movement and cumulative-gradient magnitude are proposed as practical diagnostics for whether an auxiliary token is actually learning.","Using more than one distinct thinking-token embedding produces clearer gradients, so token diversity is a direct, testable modification."],"supporting_citations":[{"why":"Introduces Thinking Tokens, the method the paper evaluates.","marker":"(Herel and Mikolov, 2023)"},{"why":"Defines Chain-of-Thought prompting, the baseline TT is compared against.","marker":"(Wei et al., 2022a)"},{"why":"Supplies the synthetic digit-multiplication dataset with intermediate calculation steps.","marker":"(Malach et al., 2024)"},{"why":"Provides the Llama 3.2 1B model used in the natural-language experiments.","marker":"(Dubey et al., 2024)"},{"why":"Provides CoT training data for GSM8k and OpenBookQA through ThoughtSource.","marker":"(Ott et al., 2023)"},{"why":"Prior pause-token work whose underperformance finding the paper's results echo.","marker":"(Goyal et al., 2024)"}],"fun_headline_variants":["Why thinking tokens fail: one shared embedding","Thinking tokens underperform: single embedding is the culprit","Two embeddings fix thinking tokens' noisy gradients","Thinking tokens lag CoT due to one-token design","Shared embedding ruins thinking tokens' potential"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on the assumption that a token embedding that barely moves and accumulates little gradient is failing because of gradient noise, rather than because the model never needs to use the token at all.","fun_headline_variants_meta":{"raw":{"variants":["Why thinking tokens fail: one shared embedding","Thinking tokens underperform: single embedding is the culprit","Two embeddings fix thinking tokens' noisy gradients","Thinking tokens lag CoT due to one-token design","Shared embedding ruins thinking tokens' potential"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000144,"raw_usage":{"total_tokens":1074,"prompt_tokens":741,"completion_tokens":333,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":357,"completion_tokens_details":{"reasoning_tokens":263}},"tokens_in":357,"tokens_out":333,"duration_ms":3816,"temperature":1.0,"reasoning_tokens":263,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:35:33.569731+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a single-thinking-token model with an auxiliary objective that forces the token's embedding to receive large, consistent gradients (for instance, by making the token predict the next reasoning step) and check whether accuracy on GSM8k rises to CoT levels; if accuracy stays flat while the embedding moves, the noisy-gradient story is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the synthetic digit-multiplication dataset with intermediate calculation steps."},{"cited_title":"ThoughtSource: A central hub for large language model reasoning data","cited_arxiv_id":"2301.11596","evidence_quote":"Provides CoT training data for GSM8k and OpenBookQA through ThoughtSource."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior pause-token work whose underperformance finding the paper's results echo."}],"review_version":1}