Pith. sign in

REVIEW 5 major objections 5 minor 23 references

State Stream Transformer (SST) : Emergent Metacognitive Behaviours Through Latent State Persistence

T0 review · 5 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A sliding-window cache of feed-forward states is claimed to let a frozen 8B instruction-tuned transformer reason better and show self-monitoring behaviours.

desk verdict A simple, concrete inference-time cache mechanism with a central claim that is not supported: the benchmark gains are confounded with extra recursive compute and tuned hyperparameters, and the headline GSM-8K number is internally inconsistent. read the letter →

arxiv 2501.18356 v1 pith:BQC5GTMG submitted 2025-01-30 cs.LG cs.AIcs.CL

Thea Aviss This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords StateStreamTransformerlatentpersistencefeed-forwardnetworkcachefrozenpretrainedweightszero-shotreasoningGSM-8KARCChallengemetacognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Current transformer decoders rebuild their internal feed-forward (FFN) states from the KV cache at every token, so no latent computation carries over from one generated token to the next. The State Stream Transformer (SST) changes that with a post-attention sliding-window cache: each block keeps a weighted, RMS-normalised copy of its previous FFN output and blends it with the current hidden state, then lets the whole stack recur two to four times per token before emitting the next token. Using the same frozen pretrained weights with no additional training, the paper reports 89.01% zero-shot accuracy on GSM-8K and 91.04% zero-shot CoT on ARC Challenge, above its base-architecture controls. It also reports qualitative outputs in which the model appears to monitor its own processing, correct an internal miscount, and reason about hypothetical and ethical scenarios in ways the base architecture does not. The paper's claim is that persistent latent-state computation enables these changes, and that the state stream, not just extra forward passes, is doing the work.

What carries the argument

The load-bearing mechanism is the latent state cache inside each transformer block. After attention, each block computes $h_{\text{blend}} = h(1-\alpha) + \mathrm{RMSNorm}(C_{t-1})[: , :s]\alpha$, where $C_{t-1}$ is the previous cached FFN output, $\alpha$ is the state-stream strength (around 0.027 in the operating regime), and the slice keeps the sequence length aligned. The blended tensor goes through the FFN, whose output becomes the new cache $C_t$. During generation the paper adds $r$ recursive full-stack forward passes per token while the KV cache is frozen, giving the latent states extra 'resolution time' before the next token is committed; $r=2$ to $4$ is the reported regime, with 2 used for initial runs and 4 for retrying failures. The cache is RMS-normalised before blending to stop magnitude explosion, is detached from the computation graph, and resets between sequences. Together these pieces create the three flows the paper describes: token-space flow, internal temporal flow through evolving caches, and vertical layer-to-layer processing.

What would settle it

Give the unmodified base architecture the same recursive treatment, with an identical frozen KV cache, the same number of forward passes per token, and the same retry-wrong-answers-at-4-recursions protocol, but with the state cache blending disabled (equivalently $\alpha = 0$). If it reaches 89.01% on GSM-8K and 91.04% on ARC Challenge, or produces similar self-referential stutter commentary, the claim that state persistence drives the effect is falsified. The paper already attempts a repetition control on the base model for qualitative outputs, but no compute-matched quantitative control is reported.

Watch

Extended reading notes

Core claim

The central claim is that a transformer needs a continuous 'state stream' to reason well: a standard transformer discards its intermediate FFN states after each token and rebuilds them from the KV cache, whereas a model that maintains and evolves those states can reason with a continuously updating internal context. The paper asserts that with exactly the same frozen weights, the SST architecture alone changes both the internal dynamics and the outputs of the model, yielding the benchmark numbers above and generating apparent metacognitive behaviours, including self-correction during a letter-counting task, commentary about its own stalling or repetition before an attractor state, planning-oriented engagement with a hypothetical teaching scenario, and stepwise ethical reasoning on a trolley problem while still refusing concrete harmful instructions. The author's preferred explanation is abductive: the qualitative behaviours and quantitative gains are best explained by the persistent state stream enabling a form of higher-order processing, rather than by noise, stochastic sampling, or learned response patterns. The evidence offered includes deterministic greedy decoding, an artificial-repetition control on the base model, and state-dynamics visualisations that show the base model resetting its FFN states between tokens while SST shows intermediate computational steps.

Load-bearing premise

The load-bearing premise is causal attribution: that the benchmark gains and metacognitive-looking outputs come from the persistent state cache itself rather than from the two to four extra recursive forward passes per token and from state-strength values tuned on the same benchmarks, since the base model is never run with the same compute or retry budget.

Editorial extensions

If this is right

  • If the claim is right, reasoning gains can be added to already-trained decoder models by changing only the inference forward pass, with no gradient updates or weight modifications.
  • Zero-shot reasoning on GSM-8K would outperform explicit 8-shot chain-of-thought prompting on the same frozen weights, meaning prompt-provided reasoning chains are not necessary and may even constrain latent reasoning.
  • The finding that recursion count did not need to grow with context length (tested to 8192 tokens) suggests the per-token compute overhead of the state stream stays roughly constant as sequences lengthen.
  • The observed 'overthinking' penalty at higher recursion counts implies that reasoning quality has an optimum resolution time that varies by task type, so fixed compute budgets are not automatically better.
  • If the safety-related observations generalise, persistent latent computation could change how instruction-tuned models balance abstract ethical reasoning against categorical refusals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the headline accuracies are a two-attempt score, since problems wrong at 2 recursions are retried at 4, so the gain over the base model must be compared at a matched inference budget before concluding that persistence beats compute.
  • Editorial inference: a decisive control would run the base architecture with the same recursion schedule and retry protocol but with cache blending set to strength zero; matching SST's scores would shift the cause from state persistence to extra forward passes.
  • Editorial inference: if the effect transfers, any compatible frozen decoder with FFN hidden states should show at least a partial gain from a sliding-window cache, a direct generalisation the paper leaves untested.
  • Editorial inference: the metacognitive reading is one interpretation of systematic stutters and self-referential comments; the paper's single repetition-control on the base model is suggestive but not exhaustive, so more permutation tests of repetition patterns would sharpen whether the apparent state awareness is architecture-specific.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes the State Stream Transformer (SST), a transformer variant that stores a weighted-decay sliding-window cache of per-layer FFN outputs and blends it into subsequent forward passes (Sec. 3.3). Using unmodified frozen Llama 3.1 8B Instruct weights, the authors compare base and SST architectures under greedy sampling, report qualitative 'metacognitive' outputs, and report 89.01% accuracy on GSM-8K (0-shot) and 91.04% on ARC Challenge (0-shot CoT). The central claim is that the persistent latent state cache alone causes the observed benchmark improvements and emergent higher-order behaviours.

Significance. If established, the finding would be significant because it suggests that an inference-time architectural change with no training can unlock latent reasoning abilities in frozen pretrained models. The paper has some strengths: identical frozen weights are used for both architectures, greedy decoding and fixed seeds provide determinism, and cross-GPU reproducibility checks are reported. However, the causal attribution to the state cache is not established. The evaluation varies recursive compute and retry policy together with alpha, and the headline GSM-8K count is internally inconsistent, so the current quantitative evidence cannot support the paper's claims.

major comments (5)
  1. [Secs. 3.4, 6.1; Table 1] The comparison in Table 1 is confounded with additional inference compute and retries: SST runs r = 2-4 recursive forward passes per token (Sec. 3.4), and the error-resolution phase in Sec. 6.1 retries every initially incorrect answer with 4 recursions, while the base architecture receives one forward pass and no retries. A base model given the same recursion count and retry protocol, together with an SST variant with alpha = 0 (or with the cross-token cache reset each step), is needed to attribute the gains to the state-cache mechanism; without such controls the headline gains are equally consistent with 'more compute and retries improve answers'.
  2. [Secs. 3.6, 6.1] The benchmark protocol is tuned on the target benchmarks: alpha and the recursion counts (2 initial, 4 on retry) are chosen from the ranges reported in Sec. 3.6, and no held-out development set or sensitivity analysis is described. Because the error-resolution phase in Sec. 6.1 is applied to the same GSM-8K and ARC test sets whose final scores are reported, the reported accuracies are post-hoc selections rather than predictions; the authors should report a full grid over alpha and recursion count, specify the selection rule, and evaluate on a held-out split.
  3. [Sec. 6.2] Sec. 6.2 contains an internal arithmetic inconsistency in the headline result: 151 incorrect out of 1,319 gives 88.55%, not the reported 89.01%; the same paragraph says 224 initial errors, of which 80 were corrected, implying 144 final errors and 89.08% accuracy, and '80 of these errors (53.0%)' is inconsistent with 224 as the base. The ARC numbers have a similar mismatch (105 wrong from 1,172 is 8.96%, while 160 initial errors minus 57 corrections leaves 103 wrong, not 105). These inconsistencies make the quantitative evidence unreliable on its own terms and must be corrected before the benchmark claims can be assessed.
  4. [Secs. 4.2, 5] The claim that the state stream is 'solely' responsible (Abstract and Sec. 5) is not supported by any ablation that removes cross-token persistence while preserving the recursive passes. The repetition-simulation control in Sec. 4.2 only tests whether artificial token repetitions in the base model elicit metacognitive language; it does not manipulate the state cache. The Fig. 4 observation that recursions do not change base-model states is expected in a deterministic base without a cache and does not demonstrate that SST state evolution is caused by the cache; a within-SST ablation (e.g., alpha = 0 or cache reset, with identical recursions and retries) is required.
  5. [Secs. 4.2-4.5] The qualitative metacognitive evidence is post hoc: categories such as 'state awareness' and 'introspective context' are defined from selected outputs and then rediscovered in those outputs, with no pre-registered criteria, task-dependent behavioral test, or quantitative measure. The paper itself notes in Sec. 4.1 that statistically valid claims about introspective context would require further research, yet the conclusions depend on those claims; a falsifiable operationalization of the proposed metacognitive behaviours is needed.
minor comments (5)
  1. [Table 1] Table 1 mixes evaluation conditions (base 8-shot CoT official vs SST 0-shot, and base 0-shot CoT 'our tests' vs SST 0-shot CoT); please present one common protocol for all rows and state all hyperparameters used in each condition.
  2. [General] No code or data availability statement is provided; given the paper's reproducibility claims, releasing evaluation scripts, seeds, and generated outputs would materially strengthen the paper.
  3. [Figures 2-4] Figures 2-4 would benefit from quantitative summaries (e.g., autocorrelation, state-difference norms) rather than qualitative visual inspection; the captions currently overstate what the figures establish.
  4. [References] Reference [19] appears in the bibliography but is not cited in the text.
  5. [Conclusion] The conclusion contains a grammatical error ('this its latent state processing') and several other typos; a careful proofread is needed.

Circularity Check

3 steps flagged · score 6.0 of 10

Reported SST gains are not independent predictions: alpha and recursion counts are tuned on the same benchmarks, failures are retried with extra compute, and 'metacognition' is defined to include the very output patterns offered as evidence.

  1. fitted input called prediction [Sec. 6.1–6.2 (Evaluation Methodology and GSM-8K results)]
    "All incorrect answers retried with 4 recursions per token - No context preserved from previous attempts - Each retry treated as an independent test ... Initially, with 2 recursions per token, the model produced 224 incorrect answers (17.0% error rate). However, when these incorrect answers were retried in isolation with 4 recursions, 80 of these errors (53.0%) were corrected."

    The claimed 89.01% GSM-8K accuracy is not the output of a fixed SST configuration; it is the final result after every initial failure was rerun on the same benchmark with a different hyperparameter (4 recursions). The retry decision is made from the test answers themselves, so the reported 'prediction' incorporates information from the target it is supposed to predict. The paper's justification that this tests 'state resolution time' does not change the fact that the evaluation protocol selects, on the test set, which items get extra compute before the score is reported.

  2. fitted input called prediction [Sec. 3.6 (Implementation Details) and Sec. 6.1 (Evaluation Methodology)]
    "State stream strength α can range freely with the most remarkable behaviours between 0.013 (1.3%) to 0.04 (4%), with optimal performance typically observed at 0.027 (2.7%). Recursion count between 2-4 iterations, adjusted based on task complexity."

    The 'optimal' α ≈ 0.027 and recursion counts 2–4 are chosen after inspecting the same benchmarks on which the paper then reports the SST's superior accuracy. Because the evaluation protocol is built from these same-benchmark observations, the concluding '89.01% / 91.04%' numbers are not tests of a first-principles architecture; they are the selected operating points of a tuned system. No held-out tuning set or sensitivity analysis is provided to show that the result does not depend on this selection.

1 more flagged steps
  1. self definitional [Sec. 1 (Introduction) and Sec. 4.2 (Introspection Behaviour Analysis)]
    "Note that metacognition, as discussed in this paper, we define as 'thinking about thinking' - specifically, the model demonstrating observable processing patterns about its own processing patterns, evidenced by output behaviours rather than simple self-description. ... aligning with our metacognitive evidence framework where we observe processing patterns about processing patterns rather than mere self-description."

    The central qualitative discovery of 'emergent metacognitive behaviours' is guaranteed by the paper's own definition: metacognition is stipulated to be 'observable processing patterns about its own processing patterns,' and the cited evidence is exactly that—stutters and self-interruptions in the output ('I think I think I think I'm getting... confused'). The observation is then presented as confirming the definition, so the 'metacognitive' classification is a labeling decision, not an independent empirical result. The repetition-simulation control rules out one alternative mechanism but does not break this definitional circularity.

full rationale

The paper's central attribution—that latent state persistence, and it alone, produces the benchmark gains and metacognitive behaviors—is not supported by an independent derivation. The quantitative claims reduce to a test-set-adaptive evaluation: α and recursion counts are selected after inspecting the same benchmarks (Sec. 3.6, 6.1), failures are retried with 4 recursions on the same GSM-8K/ARC test sets (Sec. 6.1–6.2), and the final accuracy is then presented as the architecture's standalone performance. There is no ablation that removes cross-token state persistence while keeping the recursive passes, so 'state stream' and 'extra compute plus tuned residual perturbation plus retries' are fully confounded; this is a serious validity problem, though it is closer to confounding than to definitional circularity. The qualitative metacognition claim, by contrast, is circular in the stricter sense: the paper defines metacognition as observable processing patterns about processing patterns, then offers stuttering and self-interruption as evidence of that same stipulated category. No load-bearing self-citation chain is present; all references are external, so the uniqueness/self-citation patterns do not apply. Overall, the evaluation and category definitions make the two central claims partly self-confirming, warranting a score of 6 rather than a clean non-finding.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on several unproven assumptions: that frozen Llama weights behave as a valid base for a different architecture, that alpha and recursion counts need not be independently predicted, that qualitative output patterns are valid evidence for metacognition, and that deterministic greedy runs are representative. The free parameters are tuned on the same benchmarks that are later reported as results.

free parameters (3)
  • alpha (state_stream_strength) = optimal around 0.027, range 0.013-0.04
    Tuned by the author across tasks; no independent prediction, and it is the core mechanism's strength.
  • recursion count per token = 2-4 depending on task; 2 for initial pass, 4 for retries
    Chosen per benchmark and per task; the paper reports that 4 recursions often improve results, but this is post-hoc.
  • retry protocol (retry all incorrect answers with 4 recursions)
    A selective extra-compute evaluation rule applied only to errors; inflates final accuracy relative to a single-pass evaluation.
assumptions (4)
  • domain assumption Frozen Llama 3.1 8B Instruct weights are a valid substrate for the SST architecture; hot-swapping them into a new computational graph does not invalidate the comparison.
    Used throughout all experiments; the paper assumes the same weights instantiated in the new architecture are comparable to the base.
  • ad hoc to paper The weighted-decay blend h_blend = h(1-alpha) + RMSNorm(C_{t-1}) alpha implements 'latent state persistence' rather than a small arbitrary perturbation.
    Sec 3.3 defines the mechanism; the interpretive step from this formula to 'processing continuity' is assumed.
  • domain assumption Deterministic greedy decoding with temperature 0 and a fixed seed removes stochasticity, so output differences are purely architectural.
    Sec 4.1 'token-perfect' reproduction; used to justify causal attribution.
  • ad hoc to paper Metacognitive behavior can be inferred from text patterns such as self-referential repetition commentary without a task-dependent behavioral test.
    Sec 4.1-4.2 label stutters and meta-commentary as state awareness; this is an interpretive assumption.
invented entities (1)
  • emergent metacognitive behaviors / higher-order state awareness
    purpose: Explains the qualitative outputs and benchmark gains as evidence of the model monitoring its own processing.
    The only evidence is the same outputs used to define the category; no independent, falsifiable prediction is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of State Stream Transformer (SST) : Emergent Metacognitive Behaviours Through Latent State Persistence." pith.science (2026). https://pith.science/paper/BQC5GTMG

@misc{pith2026250118356,
  author       = {Pith},
  title        = {Pith review of: State Stream Transformer (SST) : Emergent Metacognitive Behaviours Through Latent State Persistence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQC5GTMG}},
  note         = {Machine review of arXiv:2501.18356}
}
read the original abstract

We introduce the State Stream Transformer (SST), a novel LLM architecture that reveals emergent reasoning behaviours and capabilities latent in pretrained weights through addressing a fundamental limitation in traditional transformer models: the lack of latent computational continuity across autoregressive generations in the state space. SST introduces a sliding window latent state (FFN) cache with weighted decay that maintains and evolves persistent latent processes throughout autoregressive generations. Through controlled experiments comparing base and SST architectures using the same frozen weights, we demonstrate that this architectural modification alone enables enhanced reasoning capabilities which appear best explained by some form of potential higher-order processing, as evidenced by emergent metacognitive behaviours. These behaviours persist under controlled conditions designed to eliminate confounding factors such as stochastic variation or learned response patterns. Analysis of latent state distributions and processing dynamics provides evidence that it is solely the 'state stream' that is responsible for these phenomena. In quantitative evaluations, the SST achieves substantial performance improvements over the base model on two reasoning benchmarks, reaching 89.01\% accuracy on GSM-8K (0-shot) and 91.04\% on ARC Challenge (0-shot CoT). These findings indicate that persistent computation in the latent state space enables fundamentally different information processing and internal reasoning strategies, with implications for our understanding of artificial intelligence systems.

Figures

Figures reproduced from arXiv: 2501.18356 by the authors.

Figure 1
Figure 1. Block diagram of the State Stream Transformer [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. FFN output state time series visualisation comparing architectures during 100-token generation sequence. [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗
Figure 3
Figure 3. Top row: SST at 2.7% strength with 4 additional recursions, showing evolution across recursion steps (a-d) [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 4 canonical work pages

  1. [1]

    Will we run out of data? Limits of LLM scaling based on human-generated data,

    P. Villalobos, A. Ho, J. Sevilla, T. Besiroglu, L. Heim, M. Hobbhahn, "Will we run out of data? Limits of LLM scaling based on human-generated data," arXiv preprint arXiv:2211.04325v2 , Jun. 2024. Available: https://arxiv.org/abs/2211.04325

    arXiv 2024

  2. [2]

    Scaling Laws Do Not Scale

    F. Diaz, M. Madaio, "Scaling Laws Do Not Scale," arXiv preprint arXiv:2307.03201v2, Jul. 2024. Available: https://arxiv.org/abs/2307.03201

    work page Pith review arXiv 2024

  3. [3]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, D. Zhou, "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," arXiv preprint arXiv:2201.11903, Jan. 2022. Available: https://arxiv.org/abs/2201.11903

    arXiv 2022

  4. [4]

    How Reasoning Works,

    OpenAI, "How Reasoning Works," OpenAI Platform Documentation, Sept. 2024. [Online]. Accessed: Dec. 27,

    work page 2024

  5. [5]

    Emergent Representations of Program Semantics in Language Models Trained on Programs,

    C. Jin, M. Rinard, "Emergent Representations of Program Semantics in Language Models Trained on Programs," arXiv preprint arXiv:2305.11169v3, Aug. 2024. Available: https://arxiv.org/abs/2305.11169

    arXiv 2024

  6. [6]

    Do Large Language Models Latently Perform Multi-hop Reasoning?

    S. Yang, E. Gribovskaya, N. Kassner, M. Geva, S. Riedel, "Do Large Language Models Latently Perform Multi-hop Reasoning?" arXiv preprint arXiv:2402.16837, 2024

    arXiv 2024

  7. [7]

    Reasoning with Language Model is Planning with World Model,

    S. Hao, Y . Gu, H. Ma, J. Hong, Z. Wang, D. Wang, Z. Hu, "Reasoning with Language Model is Planning with World Model," arXiv preprint arXiv:2305.14992, 2023. 24

    arXiv 2023

  8. [8]

    Implicit Chain of Thought Reasoning via Knowledge Distillation,

    Y . Deng, K. Prasad, R. Fernandez, P. Smolensky, V . Chaudhary, S. Shieber, "Implicit Chain of Thought Reasoning via Knowledge Distillation," arXiv preprint arXiv:2311.01460 , Nov. 2023. Available: https://arxiv.org/abs/2311.01460

    arXiv 2023

Show all 23 references
  1. [9]

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces,

    A. Gu, T. Dao, S. Ermon, et al., "Mamba: Linear-Time Sequence Modeling with Selective State Spaces," arXiv preprint arXiv:2312.00752v2, Jan. 2024. Available: https://arxiv.org/abs/2312.00752

    2024 arXiv

  2. [10]

    Coconut: Combining Pattern-Based and Token-Free Reasoning in Language Models,

    S. Hao, S. Sukhbaatar, et al., "Coconut: Combining Pattern-Based and Token-Free Reasoning in Language Models," arXiv preprint arXiv:2412.06769v2, Dec. 2024. Available: https://arxiv.org/abs/2412.06769

    2024 arXiv

  3. [11]

    Deliberation in Latent Space via Differentiable Cache Augmentation,

    L. Liu, J. Pfeiffer, J. Wu, J. Xie, A. Szlam, "Deliberation in Latent Space via Differentiable Cache Augmentation," arXiv preprint arXiv:2412.17747, Dec. 2024. Available: https://arxiv.org/abs/2412.17747

    2024 arXiv

  4. [12]

    MemoryBank: Enhancing Large Language Models with Long-Term Memory,

    W. Zhong, L. Guo, Q. Gao, H. Ye, Y . Wang, "MemoryBank: Enhancing Large Language Models with Long-Term Memory," arXiv preprint arXiv:2305.10250v3, May 2023. Available: https://arxiv.org/abs/2305.10250

    2023 arXiv

  5. [13]

    Improving language models by retrieving from trillions of tokens,

    S. Borgeaud, et al., "Improving language models by retrieving from trillions of tokens," arXiv preprint arXiv:2112.04426, Dec. 2021. Available: https://arxiv.org/abs/2112.04426

    2021 arXiv

  6. [14]

    Universal Transformers,

    M. Dehghani, S. Gouws, O. Vinyals, et al., "Universal Transformers," International Conference on Learning Representations (ICLR), May 2019. Available: https://arxiv.org/abs/1807.03819

    2019 arXiv

  7. [16]

    Meta-llama/Llama-3.1-8B-Instruct,

    Meta AI, "Meta-llama/Llama-3.1-8B-Instruct," HuggingFace, Dec. 2024. Available: https://huggingface.co/meta- llama/Llama-3.1-8B-Instruct

    2024

  8. [17]

    Available: https://arxiv.org/abs/2407.21783

    arXiv

  9. [18]

    Root Mean Square Layer Normalisation,

    B. Zhang, R. Sennrich, "Root Mean Square Layer Normalisation," arXiv preprint arXiv:1910.07467, Oct. 2019. Available: https://arxiv.org/abs/1910.07467

    1910 arXiv

  10. [19]

    Attention Is All You Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, "Attention Is All You Need," arXiv preprint arXiv:1706.03762, Dec. 2017. Available: https://arxiv.org/abs/1706.03762

    2017 arXiv

  11. [20]

    LoRA: Low-Rank Adaptation of Large Language Models,

    E. J. Hu, Y . Shen, P. Wallis, et al., "LoRA: Low-Rank Adaptation of Large Language Models,"arXiv preprint arXiv:2106.09685v2, Oct. 2021. Available: https://arxiv.org/abs/2106.09685

    2021 arXiv

  12. [21]

    Guiding Language Model Reasoning with Planning Tokens,

    X. Wang, L. Caccia, O. Ostapenko, X. Yuan, W. Yang Wang, A. Sordoni, "Guiding Language Model Reasoning with Planning Tokens," arXiv preprint arXiv:2310.05707, 2023

    2023 arXiv

  13. [22]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, O. Tafjord, "Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge," arXiv preprint arXiv:1803.05457, Mar. 2018. Available: https://arxiv.org/abs/1803.05457 25

    2018 arXiv

  14. [2021]

    Available: https://arxiv.org/abs/2110.14168

    arXiv

  15. [2024]

    Available: https://platform.openai.com/docs/guides/reasoning/how-reasoning-works

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.