Rethinking Thinking Tokens: Understanding Why They Underperform in Practice

David Herel; David Valente; Ponnurangam Kumaraguru; Sreeram Vennam

REVIEW 4 major objections 6 minor 1 cited by

Rethinking Thinking Tokens: Understanding Why They Underperform in Practice

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Thinking Tokens, which insert a learnable token to give a model extra computation before answering, only marginally improve reasoning and consistently underperform Chain-of-Thought prompting; the paper argues the cause is the single…

desk verdict Useful negative result on Thinking Tokens, but the noisy-gradient root cause is asserted rather than tested because no two-token accuracy is reported. read the letter →

arxiv 2411.11371 v1 pith:7P7BDQZR submitted 2024-11-18 cs.CL cs.LG

Sreeram Vennam , David Valente , David Herel , Ponnurangam Kumaraguru This is my paper · ORCID

classification cs.CLcs.LG

keywords ThinkingTokensChain-of-Thoughtpromptingreasoninginlanguagemodelsgradientanalysistokenembeddingsnoisygradientsunsupervisedarithmetic

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to explain why Thinking Tokens, an unsupervised alternative to Chain-of-Thought prompting, underperform in practice. Its central claim is that the underperformance is not a failure of latent reasoning as an idea but of the mechanism: a single shared token embedding is reused across many contexts, so it receives inconsistent learning signals and noisy gradients. The authors support this with controlled comparisons on arithmetic and natural-language reasoning benchmarks and with gradient analyses showing that one thinking-token embedding barely moves during training, while two distinct token embeddings receive clear gradients and move substantially. If the claim is right, the path forward is richer, more flexible latent vectors rather than abandoning unsupervised reasoning tokens.

What carries the argument

The load-bearing object is the thinking-token embedding $e_{\mathrm{TT}}$, a single shared vector inserted as an intermediate token in the sequence. The argument is formalized as $\Delta e_{\mathrm{TT}} = \sum_{i} \nabla L(e^i_{\mathrm{TT}})$: the update to this one vector is the sum of gradient contributions from every reasoning step in which it appears, and because those steps demand different roles, the sum is noisy and the cumulative gradient is small. The contrast case is CoT, where each step has a distinct token $e^m_{\mathrm{CoT}}$ with its own isolated gradient $\nabla L(e^m_{\mathrm{CoT}})$. The mechanism is tested by comparing embedding movement and cumulative-gradient magnitude for one versus two thinking tokens.

What would settle it

Train a single-thinking-token model with an auxiliary objective that forces the token's embedding to receive large, consistent gradients (for instance, by making the token predict the next reasoning step) and check whether accuracy on GSM8k rises to CoT levels; if accuracy stays flat while the embedding moves, the noisy-gradient story is wrong.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that Thinking Tokens marginally improve accuracy over a no-reasoning baseline and consistently underperform Chain-of-Thought reasoning on digit multiplication, GSM8k, and OpenBookQA, and that this gap is caused by the single-token embedding. Because the same embedding vector is reused at every internal reasoning step, the learning signal it receives averages over conflicting roles, producing high-variance, low-magnitude gradient updates; the embedding therefore stays near its initialization. Introducing two distinct thinking-token embeddings changes the gradient pattern visibly, which the paper takes as empirical validation of the hypothesis.

Load-bearing premise

The argument rests on the assumption that a token embedding that barely moves and accumulates little gradient is failing because of gradient noise, rather than because the model never needs to use the token at all.

Editorial extensions

If this is right

Thinking Tokens as currently implemented should not be expected to replace Chain-of-Thought prompting on tasks that require structured, multi-step reasoning.
Adding Thinking Tokens to a CoT pipeline contributes little: the TT+CoT results track CoT alone, so the extra token is effectively inert.
The single-embedding design, not latent reasoning in general, is the identified bottleneck; methods that give the model a richer internal representation should be the target of future work.
Embedding movement and cumulative-gradient magnitude are proposed as practical diagnostics for whether an auxiliary token is actually learning.
Using more than one distinct thinking-token embedding produces clearer gradients, so token diversity is a direct, testable modification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the same failure mode likely applies to any dummy token inserted without an explicit training signal: if nothing forces the model to rely on the token, low gradient flow may simply mean the model learned to ignore it, and the causal role of 'noise' remains open.
A testable extension of the paper's logic is that a continuous 'thinking vector' or per-step embeddings, rather than one shared discrete token, should close much of the gap with CoT on arithmetic benchmarks; this follows from the paper's claim but is not itself tested here.
The two-token result suggests that the benefit comes from giving gradient descent distinct coordinates to push on, not from more thinking time; an experiment that increases the number of thinking steps with a single embedding and observes no accuracy gain would sharpen this conclusion.
If the noisy-gradient explanation is correct, then enlarging the embedding dimension or adding an auxiliary prediction loss on the thinking token should increase cumulative gradient and, in turn, accuracy; that is a direct, falsifiable prediction the paper does not run.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a useful negative result wrapped in a root-cause claim that the experiments don't actually support. Table 1 clearly shows single Thinking Tokens underperform CoT and can hurt (GSM8k 4.51 vs baseline 6.30; OpenBookQA unchanged). That part is solid and worth knowing. The gradient analysis—one embedding barely moves, cumulative gradient small, two distinct embeddings move more—is a genuinely new diagnostic, and the two-token comparison is a reasonable falsifiable check. But the causal story is never tested. The paper never reports accuracy for the two-token variant. The fact that two embeddings receive larger gradients shows only that the optimizer updates them more, not that the model reasons better. The 'noisy gradients' conclusion is partly definitional: low cumulative gradient is used as evidence of noise, but the alternative explanation—the model learns to ignore an optional token it has no pressure to use—fits the same observations. The theoretical section is schematic; equations (2)-(4) assert inconsistency rather than deriving it. The abstract overstates the results: 'marginally improves' contradicts the GSM8k and OpenBookQA numbers. Also, no error bars or seeds are reported, which matters for small finetuning runs. That said, the paper is not incoherent. It does a controlled comparison and reports honest numbers. The two-token embedding movement is a real observation that could be turned into a proper test if the authors added a two-token condition to Table 1. I'd send this to peer review, but with the expectation that the root-cause claim be either scoped down to 'consistent with noisy gradients' or supported by a performance experiment.

Referee Report

4 major / 6 minor

Summary. The paper compares Thinking Tokens (TT), Chain-of-Thought (CoT), TT+CoT, and a baseline on digit multiplication and on GSM8k and OpenBookQA, and reports that TT underperforms CoT. It hypothesizes that this underperformance is caused by the reuse of a single TT embedding, which produces inconsistent learning signals and noisy gradients. To test this, the authors measure embedding movement and cumulative gradient magnitude for one- and two-token variants, finding that two distinct TT embeddings move more and receive larger cumulative gradients. The paper concludes that TT's reliance on a single embedding is the root cause of its poor performance.

Significance. If the empirical comparison is reliable, the paper provides a useful negative result: an unsupervised 'thinking token' mechanism does not match explicit step-by-step supervision, and in some settings it can degrade performance. The gradient/embedding measurements are an appropriate way to probe what the model learns about a newly introduced token, and the two-token condition is a falsifiable intervention. However, the central causal claim—that noisy gradients from a single shared embedding explain the underperformance—is not tested end-to-end, because no accuracy result is reported for the two-token variant. The current significance is therefore mainly empirical (the negative result), not mechanistic.

major comments (4)

[Abstract and Table 1] The abstract states that Thinking Tokens 'marginally improves performance,' but Table 1 shows that TT reduces GSM8k exact match relative to baseline (4.51 vs 6.30) and produces no change on OpenBookQA (37.2 vs 37.2). The abstract therefore contradicts the paper's own main table and should be corrected.
[§5.1, Figures 4–5] The causal claim that a single embedding causes underperformance through noisy gradients is not tested. The two-token variant shows larger embedding movement and larger cumulative gradients, but no accuracy numbers are reported for the two-token condition. Without showing that using two distinct TT embeddings improves task performance (or at least does not degrade it), the evidence does not establish that the single embedding is the cause of the underperformance; the improved gradient statistics could simply reflect that the optimizer updates a new token more, without affecting reasoning quality.
[§3, Eq. (4) and §4.2] The theoretical basis for 'noisy gradients' is asserted rather than derived. Equation (4) merely sums per-step gradient contributions; it does not show that the per-step signals have zero mean or that role ambiguity translates into gradient noise. Moreover, the operational definition of noise in §4.2—a low cumulative gradient—is partly definitional: a token that the model largely ignores would also have a small cumulative gradient. The paper should either provide a direct measure of gradient cancellation (e.g., per-step cosine similarity or variance) or temper the claim that low cumulative gradient is evidence of noise.
[§4.1.1 and Appendix A.1.1] The main text says the digit-multiplication model is trained for 100 epochs, while the appendix says 50 epochs. This inconsistency affects reproducibility. Furthermore, no standard deviations, confidence intervals, or multiple-seed results are reported, which is particularly problematic for the small-gap comparisons (e.g., GSM8k TT vs baseline and OpenBookQA TT vs baseline) that the abstract describes as 'marginal.'

minor comments (6)

[Abstract] There are grammatical errors in the first sentence, including 'TTs marginally improves performance,' which should be 'marginally improve.'
[Figures 2–5] The x-axis labels in all four figures appear garbled ('x = ( ) + b e p u t' and 'x = ( ) + b e p u t ts'); the figures should be re-generated with readable token labels.
[Table 1] The entry '> 0' for 4-digit multiplication is unsatisfying; the paper should report the actual accuracy or explain why the run was terminated and how that affects the comparison.
[§6 Discussion] The claim that TT+CoT 'perform[s] as if they weren't present' is inaccurate for digit multiplication, where TT+CoT slightly exceeds CoT (92.3 vs 91.9 for 2-digit, 67.8 vs 66.3 for 3-digit); this should be acknowledged.
[Appendix A.1.2] The text says 'cite as lrfinder' for the learning-rate finder; this placeholder should be replaced with a proper citation or description.
[§4.2] The two-token experiment is described only briefly; details about how the two-token model is trained, how many tokens are inserted, and whether it is evaluated on the same benchmarks should be provided.

Circularity Check

1 steps flagged · score 4.0 of 10

Root-cause mechanism is partly definitional: 'noisy gradients' are measured by the same low cumulative gradient used as evidence, while the benchmark comparison itself is independent.

self definitional [Section 4.2 (Gradient Analysis) and Section 5.1 (Gradient Analysis)]
"Noisy Gradients We calculate how far the embedding travels. If the embedding hasn't moved much, during training, that could be a strong indicator of noisy gradients. We also calculate the cumulative gradient. Noisy gradients result in a low cumulative gradient since the mean of noise is zero."

The paper defines 'noisy gradients' operationally as low cumulative gradient and small embedding movement, then uses those same observations to 'verify' the hypothesis that single-token embeddings produce noisy gradients. Because the proxy is chosen so that zero-mean noise implies low cumulative gradient, observing low cumulative gradient is true by definition rather than independent evidence. The two-token manipulation is a genuine falsifiable intervention, but Table 1 never reports a two-token performance condition, so the causal link from gradient statistics to task underperformance is never tested. The benchmark comparison itself is external and independent, which is why the circularity is partial.

full rationale

The central empirical finding—that Thinking Tokens underperform Chain-of-Thought—is established by Table 1 through direct benchmark comparisons and is not derived from the hypothesis, so it is not circular. The circularity is confined to the mechanistic validation: Section 4.2 operationalizes 'noisy gradients' as small embedding movement and low cumulative gradient (because zero-mean noise cancels in the cumulative sum), and Section 5.1 then cites those same measurements as confirming the hypothesis that noisy gradients cause the observed stagnation. The proxy and the predicted observation are the same quantity, making the confirmation partly definitional. The two-token experiment moves the embeddings more and yields larger cumulative gradients, but because no two-token accuracy result is reported, it never closes the gap between gradient statistics and task performance. The underperformance comparison remains a valid independent result; only the root-cause explanation is self-referential. Score 4 reflects this partial, non-central circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The root-cause argument rests on unproven equivalences between gradient statistics and learning quality; no causal experiment ties the two-token gradient improvement to task accuracy. The underperformance comparison itself is direct and needs fewer assumptions.

assumptions (4)

domain assumption Each CoT reasoning token has an isolated role and therefore provides stable gradient updates (Eq. 2).
Stated without proof in Section 3.1; used as the contrast class for TT's noisy gradients.
ad hoc to paper Reusing a shared TT embedding across steps produces ambiguous, noisy learning signals (Eq. 4).
This is the paper's hypothesis, asserted in Section 3; it is the thing the experiments are meant to validate, not an established result.
ad hoc to paper Low cumulative gradient magnitude indicates noisy gradients with zero mean.
Section 4.2 defines the metric; the paper assumes noise cancels in the cumulative sum, making this a proxy for signal quality.
ad hoc to paper Embedding movement from initialization reflects how well the token is learned.
Figures 2 and 5 are interpreted as expressivity and learning quality, but the paper does not establish that stagnation is distinct from the model simply not needing the token.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Rethinking Thinking Tokens: Understanding Why They Underperform in Practice." pith.science (2026). https://pith.science/paper/7P7BDQZR

@misc{pith2026241111371,
  author       = {Pith},
  title        = {Pith review of: Rethinking Thinking Tokens: Understanding Why They Underperform in Practice},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7P7BDQZR}},
  note         = {Machine review of arXiv:2411.11371}
}

read the original abstract

Thinking Tokens (TT) have been proposed as an unsupervised method to facilitate reasoning in language models. However, despite their conceptual appeal, our findings show that TTs marginally improves performance and consistently underperforms compared to Chain-of-Thought (CoT) reasoning across multiple benchmarks. We hypothesize that this underperformance stems from the reliance on a single embedding for TTs, which results in inconsistent learning signals and introduces noisy gradients. This paper provides a comprehensive empirical analysis to validate this hypothesis and discusses the implications for future research on unsupervised reasoning in LLMs.

Figures

Figures reproduced from arXiv: 2411.11371 by the authors.

**Figure 3.** One TT embedding receives insufficient cu [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 2.** One TT embedding hardly moves from the initialized value. 4 Experiments We first compare TT against CoT in section 4.1 on both synthetic data and popular natural language benchmarks. We perform our analysis on tasks in which intermediate computation can help neural networks learn, such as GSM8k. We then verify our hypothesis empirically in section 4.2 by monitoring both the embedding weights and the gradient of the … view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Two TT embeddings show clear deviation from initialization. 5 Analysis [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Enhancing Latent Computation in Transformers with Latent Tokens
cs.LG 2025-05 conditional novelty 5.0 of 10

Latent tokens inserted periodically during decoding, with position IDs shared with the following verbal token, improve LLM performance and OOD generalization when only the token embeddings are fine-tuned.

Reference graph

Works this paper leans on

10 extracted references · 3 canonical work pages · cited by 1 Pith paper

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024
[4]

Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. 2024. https://openreview.net/forum?id=oBpHVLzkfm Think before you speak: Training language models with pause tokens . In International Conference on Learning Representations (ICLR)

work page 2024
[5]

David Herel and Tomas Mikolov. 2023. https://arxiv.org/abs/2405.08644 Thinking tokens for language modeling . In Proceedings of the 8th Conference on Artificial Intelligence and Theorem Proving (AITP 2023)

arXiv 2023
[6]

Eran Malach et al. 2024. https://icml.cc/virtual/2024/poster/33369 Auto-regressive next-token predictors are universal learners . In Proceedings of the 41st International Conference on Machine Learning (ICML)

work page 2024
[7]

Simon Ott, Konstantin Hebenstreit, Valentin Liévin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther, and Matthias Samwald. 2023. https://doi.org/10.48550/ARXIV.2301.11596 Thoughtsource: A central hub for large language model reasoning data . arXiv preprint

work page Pith review arXiv doi:10.48550/arxiv.2301.11596 2023
[8]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022 a . https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reasoning in large language models . In Advances in Neural Information Processing Syst...

2022

Show all 10 references

[9]

Jason Wei et al. 2022 b . Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903

2022 arXiv
[10]

Xue Zhang, Yuchen Sun, et al. 2023. Towards revealing the mystery behind chain of thought: A theoretical perspective. arXiv preprint arXiv:2305.15408

2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024

[4] [4]

Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. 2024. https://openreview.net/forum?id=oBpHVLzkfm Think before you speak: Training language models with pause tokens . In International Conference on Learning Representations (ICLR)

work page 2024

[5] [5]

David Herel and Tomas Mikolov. 2023. https://arxiv.org/abs/2405.08644 Thinking tokens for language modeling . In Proceedings of the 8th Conference on Artificial Intelligence and Theorem Proving (AITP 2023)

arXiv 2023

[6] [6]

Eran Malach et al. 2024. https://icml.cc/virtual/2024/poster/33369 Auto-regressive next-token predictors are universal learners . In Proceedings of the 41st International Conference on Machine Learning (ICML)

work page 2024

[7] [7]

Simon Ott, Konstantin Hebenstreit, Valentin Liévin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther, and Matthias Samwald. 2023. https://doi.org/10.48550/ARXIV.2301.11596 Thoughtsource: A central hub for large language model reasoning data . arXiv preprint

work page Pith review arXiv doi:10.48550/arxiv.2301.11596 2023

[8] [8]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022 a . https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reasoning in large language models . In Advances in Neural Information Processing Syst...

2022