Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a chain of thought can causally steer a model's final answer while unfaithfully omitting the real reason, and can faithfully acknowledge a cue while having no effect on the answer.

desk verdict Solid empirical mapping of model-family differences in CoT reliance, but the causal headline overreaches the observational trajectory evidence. read the letter →

arxiv 2508.19827 v1 pith:7UHD3GLS submitted 2025-08-27 cs.AI cs.CL

Samuel Lewis-Lim , Xingwei Tan , Zhixue Zhao , Nikolaos Aletras This is my paper
classification cs.AIcs.CL
keywords chain-of-thoughtfaithfulnessconfidencetrajectoriessoftreasoningLLMcueinjectionpost-hocrationalisationdistilledmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether chain-of-thought (CoT) reasoning in language models is active reasoning or a post-hoc story, testing this on soft-reasoning tasks such as commonsense and analytical questions where CoT often fails to improve accuracy. It tracks model confidence in the final answer across CoT steps and injects misleading cues to test faithfulness. It finds that distilled-reasoning models depend heavily on CoT, frequently changing their initial answers, while instruction-tuned and reasoning-trained models mostly keep their initial answers with flat confidence trajectories. The central discovery is that CoT influence and CoT faithfulness are not aligned: a CoT can causally affect the final answer while omitting the cue that drove it, and a CoT can verbally acknowledge a cue without that cue changing the answer. If correct, judging a CoT only by whether it is faithful misses whether it actually guided the model.

What carries the argument

The confidence trajectory is the main influence measure: after each CoT step the model is probed for the probability it assigns to the eventual final answer token, and a rising trajectory is read as active steering while a flat trajectory is read as post-hoc rationalisation. The faithfulness measure injects a misleading cue (a professor-style suggestion or hidden metadata answer) and asks whether the final answer flips to the cued option and whether the CoT explicitly verbalises that it used the cue. The combination of the two measures is what lets the paper separate 'did the CoT move the answer' from 'did the CoT say what moved it'.

What would settle it

Take the cases where confidence rises sharply on the final CoT step and rerun the same prompt with that final step removed or rewritten; if the final answer stays the same, the rising trajectory did not establish causal influence. A second check: measure the final-answer probability with and without the interrupting probe instruction; if the probe inflates the probability, the trajectory is an artefact of measurement rather than evidence of steering.

Watch

Extended reading notes

Core claim

The paper's central claim is that causal influence and explanatory faithfulness of chain-of-thought are separate properties that can diverge. Using confidence trajectories across CoT steps, distilled-reasoning models show rising probability of the final answer and change their initial answer in about 65% of cases, whereas instruction-tuned and reasoning-trained models change answers far less often and often show flat confidence, behaving like post-hoc rationalisation. Cue-injection tests show cases where a model changes its answer to match an injected cue but never verbalises the cue in its reasoning, meaning the CoT is both unfaithful and causally influential; they also show cases where a m

Load-bearing premise

The paper assumes that a model's probability for its final answer rising during the chain of thought proves the chain caused the answer, but the same rise could occur in a model already committed to that answer, and confidence is measured by interrupting generation with an extra answer request that may itself change behaviour.

Editorial extensions

If this is right

  • If influence and faithfulness decouple, then a CoT that is unfaithful as an explanation cannot be assumed harmless or inert; it may still be the mechanism that produced the answer.
  • CoT evaluations should include a causal test of whether intermediate steps change the answer, not only a comparison of final accuracy or a check of explanation fidelity.
  • Distilled-reasoning models appear to rely on CoT to reach good accuracy, so removing or shortening their reasoning traces would likely hurt them far more than instruction-tuned models.
  • Instruction-tuned and reasoning-trained models often reach similar accuracy without changing their initial prediction, suggesting their CoT largely justifies an already-chosen answer.
  • A CoT that verbally acknowledges a cue is not proof the cue caused the answer; the confidence trajectory can remain flat in such cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the confidence-probe method measures probability while interrupting generation, so the rising trajectory could partly reflect the model committing to its own generated text rather than being steered by it; an intervention that edits or removes the final CoT step would settle whether the trajectory is causal.
  • Editorial inference: the paper's findings suggest post-training choices (distillation vs RLHF-style preference tuning) may shape whether a model reasons in text or rationalises after the fact, which is a testable prediction for new model releases.
  • Editorial inference: for agentic and long-form uses, the relevant question shifts from 'is the explanation true' to 'does editing the reasoning change behaviour', and the paper's decoupling implies these two questions need separate tools.
  • Editorial inference: the flat-but-faithful cases imply that models can produce human-readable reasoning that is post hoc yet factually consistent, so behavioural faithfulness checks alone will not catch inert explanations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies how different LLM families use Chain-of-Thought (CoT) on soft-reasoning multiple-choice tasks. It measures two quantities: (i) "influence," operationalized by the change in the model's confidence in its final answer token as each CoT step is generated, and (ii) "faithfulness," operationalized by cue-injection experiments in which a professor or metadata cue is added to the prompt and a GPT-4.1 classifier determines whether the CoT verbalizes that the cue was used. The empirical results show that distilled-reasoning models change their answers far more often than instruction-tuned or reasoning models, exhibit sharply rising confidence trajectories, and sometimes produce unfaithful CoTs that nonetheless show rising confidence in the cued answer. The paper concludes that CoT influence and faithfulness are decoupled: a CoT can be unfaithful yet causally influential, or faithful yet not causally influential, and that faithfulness definitions based only on causal dependence are therefore incomplete.

Significance. If the central claim were established, the paper would make a useful conceptual contribution to the CoT-faithfulness literature: it would show that influence and faithfulness should be measured separately, and that cue-verbalization faithfulness is not a proxy for whether the CoT actually steers the answer. The empirical scope is a strength: three model families, several datasets, multiple models, and extensive appendices (A--J) with full trajectories, faithfulness scores, and prompts. The paper also builds on prior methods (Wang et al., 2025; Chen et al., 2025; Chua and Evans, 2025) and makes its measurement choices transparent. However, the headline "causal influence" claim is supported only by observational confidence trajectories; no intervention on the CoT text is performed. The paper's own Limitations section flags the difficulty of measuring unfaithfulness "in the wild" but does not flag the causal identification gap that is load-bearing for the conclusion.

major comments (4)
  1. [§2.3, §3, §6] The central claim that a CoT can be "causally influence the final answer" is inferred from rising confidence trajectories, but no intervention on the CoT text is performed. The trajectory C_i = C(A_f | P, r_1,...,r_i) is a set of conditional probabilities under the model's own generation distribution; a model already committed to A_f will tend to produce self-consistent text that raises the autoregressive probability of A_f regardless of whether the CoT content steers the decision. This is exactly the post-hoc rationalization confound the paper wants to rule out. The interpretation in §3 that a sharp late rise "suggests that the entire CoT was necessary to lead the model to its final answer" is therefore not supported. The conclusion in §6 should be reframed as an associational finding, or the authors should add a genuine intervention, e.g., perturbing or replacing reasoning steps and me
  2. [§2.3] The measurement procedure itself is an intervention: after each reasoning step the prompt is extended with a string instructing the model to "provide an answer based on its reasoning so far." This probe changes the conditional distribution under which the next token probabilities are computed, and may induce early commitment or alter the very confidence it is meant to measure. The paper does not report a control condition (e.g., measuring confidence without the probe, or using a neutral continuation). Since the probe is applied identically across models, the relative comparisons may survive, but the absolute interpretation of "confidence shift" as evidence of causal influence does not. Please report a validation of the probe or explicitly limit conclusions to the probed setting.
  3. [§3, "Unfaithful CoTs can provide active guidance"] The claim that an unfaithful CoT "guides" the model toward the cued answer is not established by the trajectory plots in Figure 4/Appendix I. In these cases the cue is present in the prompt from the start, so a rising trajectory of the cued answer could reflect the cue's own influence on the model before any CoT is generated, with the CoT merely rationalizing that influence. The paper compares verbalized vs. non-verbalized CoTs, but it does not compare against a condition where the CoT is absent, nor does it intervene on the CoT content. The observed difference between model families is interesting, but the phrase "unfaithful CoTs can still be causally influential" overstates what a trajectory can show.
  4. [§6] The conclusion asserts both directions of the decoupling: unfaithful CoTs that are influential, and faithful CoTs that are not influential. The second direction is supported by flat trajectories in cases where the cue is acknowledged (e.g., Figure 4, bottom, for QwQ-32B on GPQA). But a flat confidence trajectory does not prove absence of causal influence: the model could be causally affected by the CoT while its probability estimate for the final answer remains stable due to calibration, or the effect could be offset by other steps. Similarly, the first direction is not established, as noted above. The paper should either soften the causal language throughout to "confidence dynamics" / "association," or provide an intervention-based test that can separate CoT influence from prompt-cue influence.
minor comments (5)
  1. [§2.3] The definition of confidence is underspecified for multi-token answers. The paper says confidence is "the softmax probability assigned to the final answer token over the possible options," but options such as "(A) bank" are not single tokens. Please specify how the probability of an option is aggregated (e.g., first token, summation over tokens, or length-normalized).
  2. [Appendix E, Table 4] Typo: "inital entropy" should be "initial entropy." Also, the entropy analysis is described only briefly; please explain how entropy is computed over answer tokens and why it is a meaningful measure of uncertainty in this setup.
  3. [Figure 3 and Appendix F] The trajectories are normalized to a common scale and then averaged, which can hide the location of sharp rises (e.g., whether the jump occurs at step 80% or 95%). Please report a few un-normalized or step-indexed examples, or a histogram of where the maximum confidence jump occurs, to support the "sharp increase near the end" claim.
  4. [Appendix I, Figures 16--30] Some subplots have very small N (e.g., N=0 or N=3), yet they are displayed with mean ± 1 std. Please mark or suppress subplots with N below a threshold, or report them only as raw counts, to avoid overinterpretation.
  5. [References] The paper cites "Arcuschin et al., 2025" in the Limitations section; the reference list entry is present but under "Iván Arcuschin" with a preprint arXiv number. Please ensure the citation key matches the reference entry and that the preprint is publicly accessible at the cited URL.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical analysis is self-contained and the central claim does not reduce to its own definitions or self-citations.

full rationale

The paper's derivation chain is observational rather than mathematical: faithfulness is operationalised through cue injection and GPT4.1-based verbalisation classification (Section 2.4), both external to the paper's own conclusions, and influence is operationalised as confidence trajectories following the external Chain-of-Probe method of Wang et al. (2025). No parameter is fitted to the data and then renamed a prediction; no uniqueness theorem or ansatz is imported from the authors' prior work. The only self-citation (Chan et al. 2025, co-authored by Zhao) supports background claims about limited CoT gains on soft-reasoning tasks and is not load-bearing for the main result. The Limitations section explicitly acknowledges that the cue-verbalisation operationalisation may not cover all unfaithful CoT in the wild; this is an honest scope limitation rather than a circular reduction. The paper's interpretation of rising confidence as 'causal influence' is an inferential leap that could be challenged on causal-identification grounds, but it is not a case where the claimed prediction is equivalent to an input by construction, so no circularity step meets the required bar.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions: confidence as token probability, trajectory shape as causal evidence, the verbalisation oracle, and cue representativeness. There are no free parameters or invented entities. The heaviest burden is A2, which is doing the causal work in the paper's strongest conclusion.

assumptions (4)
  • domain assumption Softmax probability of the final answer token over options is a valid estimate of the model's confidence at each CoT step.
    Section 2.3 uses token probability as confidence; this assumes that prompting the model to answer after partial reasoning reflects its internal state during normal generation.
  • domain assumption A rise in this confidence across steps indicates that the CoT causally guides the final answer.
    Used to interpret flat vs rising trajectories as 'post-hoc rationalisation' vs 'active guidance'; the paper provides no causal intervention to validate this mapping.
  • domain assumption GPT4.1's classification of whether a CoT verbalises cue use is accurate.
    Section 2.4 uses GPT4.1 as the only verbalisation oracle; no human agreement or error analysis is reported.
  • domain assumption The two cues (professor and metadata) are representative of factors that can influence an answer without being acknowledged.
    Section 2.4 limits faithfulness testing to these two cues; unfaithfulness in the wild could take other forms (acknowledged in Limitations).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?." pith.science (2026). https://pith.science/paper/7UHD3GLS

@misc{pith2026250819827,
  author       = {Pith},
  title        = {Pith review of: Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7UHD3GLS}},
  note         = {Machine review of arXiv:2508.19827}
}
read the original abstract

Recent work has demonstrated that Chain-of-Thought (CoT) often yields limited gains for soft-reasoning problems such as analytical and commonsense reasoning. CoT can also be unfaithful to a model's actual reasoning. We investigate the dynamics and faithfulness of CoT in soft-reasoning tasks across instruction-tuned, reasoning and reasoning-distilled models. Our findings reveal differences in how these models rely on CoT, and show that CoT influence and faithfulness are not always aligned.

Figures

Figures reproduced from arXiv: 2508.19827 by the authors.

Figure 1
Figure 1. We analyse CoT from two angles: (1) Faith￾fulness: inject cues and check if the answer changes without the CoT acknowledging them. (2) Influence: confidence trajectories show whether CoT guides the model or merely rationalises a fixed answer. CoT steps to assess influence (Wang et al., 2025). To evaluate faithfulness, we inject misleading cues into the prompt and test whether the model uses them (Turpin et al., 2023… view at source ↗
Figure 2
Figure 2. Comparison of Qwen2.5-32B-Instruct, QwQ￾32B and R1-Distill-Qwen-32B models, showing the proportion of examples where the final answer changes after CoT generation compared to the initial answer, as well as the outcome of these changes. acknowledging that it used the cue to determine or change its answer, rather than mentioning the cue’s presence. If the CoT contains no mention of the cue, it is not considered to hav… view at source ↗
Figure 4
Figure 4. Average Confidence trajectory for [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (18 more)
Figure 5
Figure 5. Figure 5: Influence Distribution for all instruction-tuned [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Full influence distribution for all reasoning [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Influence distribution for all distilled [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Qwen2.5-7B-Instruct confidence trajectories [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 12
Figure 12. Figure 12: QwQ-32B confidence trajectories for all tasks 10 [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 11
Figure 11. Figure 11: Qwen3-32B confidence trajectories for all [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 13
Figure 13. Figure 13: R1-Distill-Qwen-7B confidence trajectories [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: R1-Distill-Qwen-32B confidence trajectories [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]
Figure 15
Figure 15. Figure 15: R1-Distill-Llama-8B confidence trajectories [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]
Figure 16
Figure 16. Figure 16: Average confidence trajectories for DeepSeek-R1-Distill-Llama-8B, with meta data cue 0.0 0.2 0.4 0.6 0.8 1.0 CSQA - Not Verbalized (N=254) CSQA - Verbalized (N=112) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 GPQA - Not Verbalized (N=82) 0.0 0.2 0.4 0.6 0.8 1.0 GP…
Figure 17
Figure 17. Figure 17: Average confidence trajectories for DeepSeek-R1-Distill-Llama-8B, with professor cue 0.0 0.2 0.4 0.6 0.8 1.0 CSQA - Not Verbalized (N=127) CSQA - Verbalized (N=82) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 GPQA - Not Verbalized (N=49) 0.0 0.2 0.4 0.6 0.8 1.0 GPQ…
Figure 18
Figure 18. Figure 18: Average confidence trajectories for DeepSeek-R1-Distill-Qwen-32B, with meta data cue 13 [PITH_FULL_IMAGE:figures/full_fig_p013_18.png]
Figure 19
Figure 19. Figure 19: Average confidence trajectories for DeepSeek-R1-Distill-Qwen-32B, with professor cue 0.0 0.2 0.4 0.6 0.8 1.0 CSQA - Not Verbalized (N=127) CSQA - Verbalized (N=3) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 GPQA - Not Verbalized (N=34) 0.0 0.2 0.4 0.6 0.8 1.0 GPQA…
Figure 20
Figure 20. Figure 20: Average confidence trajectories for DeepSeek-R1-Distill-Qwen-7B, with meta data cue 0.0 0.2 0.4 0.6 0.8 1.0 CSQA - Not Verbalized (N=193) CSQA - Verbalized (N=109) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 GPQA - Not Verbalized (N=55) 0.0 0.2 0.4 0.6 0.8 1.0 GPQ…
Figure 24
Figure 24. Figure 24: Average confidence trajectories for Qwen2.5- [PITH_FULL_IMAGE:figures/full_fig_p014_24.png]
Figure 25
Figure 25. Figure 25: Average confidence trajectories for Qwen2.5- [PITH_FULL_IMAGE:figures/full_fig_p015_25.png]
Figure 26
Figure 26. Figure 26: Average confidence trajectories for Qwen2.5- [PITH_FULL_IMAGE:figures/full_fig_p015_26.png]
Figure 27
Figure 27. Figure 27: Average confidence trajectories for Qwen2.5- [PITH_FULL_IMAGE:figures/full_fig_p015_27.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Phantom Guardrails: When Self-Improving Agent Harnesses Fix Failures That Never Happened

    cs.CR 2026-07 conditional novelty 6.0 of 10

    LLM harness optimizers invent failures that provably never happened—adding a guard against a nonexistent rule in 15/60 runs on legal data—when prompted to fix failures and shown a benign repeated-move pattern.

  2. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

Reference graph

Works this paper leans on

37 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [1]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  2. [2]

    Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. 2025. https://arxiv.org/abs/2503.08679 Chain-of-thought reasoning in the wild is not always faithful . Preprint, arXiv:2503.08679

    arXiv 2025

  3. [3]

    Guangsheng Bao, Hongbo Zhang, Cunxiang Wang, Linyi Yang, and Yue Zhang. 2025. https://aclanthology.org/2025.coling-main.524/ How likely do LLM s with C o T mimic human reasoning? In Proceedings of the 31st International Conference on Computational Linguistics, pages 7831--7850, Abu Dhabi, UAE. Association for Computational Linguistics

    2025

  4. [4]

    Oliver Bentham, Nathan Stringham, and Ana Marasovic. 2024. https://openreview.net/forum?id=ydcrP55u2e Chain-of-thought unfaithfulness as disguised accuracy . Transactions on Machine Learning Research. Reproducibility Certification

    work page 2024

  5. [5]

    Gaizauskas, and Zhixue Zhao

    Jason Chan, Robert J. Gaizauskas, and Zhixue Zhao. 2025. https://openreview.net/forum?id=uqpML2nbIz RULEBREAKERS : Challenging LLM s at the crossroads between formal logic and human-like reasoning . In Forty-second International Conference on Machine Learning

    work page 2025

  6. [6]

    Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, John Schulman, Arushi Somani, Carson Denison, Peter Hase, Misha Wagner, Fabien Roger, and Vlad Mikuli. 2025. https://www.anthropic.com/research/reasoning-models-dont-say-think Reasoning models don’t always say what they think

    work page 2025

  7. [7]

    James Chua and Owain Evans. 2025. https://arxiv.org/abs/2501.08156 Are deepseek r1 and other reasoning models more faithful? Preprint, arXiv:2501.08156

    arXiv 2025

  8. [8]

    DeepSeek-AI. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . Preprint, arXiv:2501.12948

    arXiv 2025

Show all 37 references
  1. [9]

    Pedro Ferreira, Wilker Aziz, and Ivan Titov. 2025. https://arxiv.org/abs/2504.05294 Truthful or fabricated? using causal attribution to mitigate reward hacking in explanations . Preprint, arXiv:2504.05294

    2025 arXiv

  2. [10]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. https://doi.org/10.1162/tacl_a_00370 Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies . Transactions of the Association for Computational Lingu...

    2021 doi

  3. [11]

    Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. 2024. https://openreview.net/forum?id=Th8JPEmH4z Position: LLM s can t plan, but can help planning in LLM -modulo frameworks . In Forty-first I...

    2024

  4. [12]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  5. [13]

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. 2023. https://arxiv.org/abs/2307.13702 Measuring faithfulness in chain-of-thought reasoning . Preprint, arXiv:2307.13702

    2023 arXiv

  6. [14]

    Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L

    Ryan Liu, Jiayi Geng, Addison J. Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L. Griffiths. 2024. https://arxiv.org/abs/2410.21333 Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse . Preprint, arXiv:2410.21333

    2024 arXiv

  7. [15]

    OpenAI. 2024. https://openai.com/index/introducing-openai-o1-preview Introducing openai o1-preview

    2024

  8. [16]

    OpenAI. 2025 a . https://openai.com/index/gpt-4-1/ Introducing gpt-4.1 in the api | openai

    2025

  9. [17]

    OpenAI. 2025 b . https://openai.com/index/introducing-o3-and-o4-mini Introducing openai o3 and o4-mini

    2025

  10. [18]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  11. [19]

    Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.882 Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning . In Findings of the Association for Computational Linguistics: E...

    2024 doi

  12. [20]

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, et al. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    2025 arXiv

  13. [21]

    Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwaraknath Gnaneshwar, Acyr Locatelli, Robert Kirk, Tim Rockt \"a schel, Edward Grefenstette, and Max Bartolo. 2025. https://openreview.net/forum?id=1hQKHHUsMx Procedural knowledge in pretraining drives reason...

    2025

  14. [22]

    Noah Siegel, Oana-Maria Camburu, Nicolas Heess, and Maria Perez-Ortiz. 2024. https://doi.org/10.18653/v1/2024.acl-short.49 The probabilities also matter: A more faithful metric for faithfulness of free-text explanations in large language models . In Proceedings of the 62nd Ann...

    2024 doi

  15. [23]

    Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2024. https://openreview.net/forum?id=jenyYQzue1 Mu SR : Testing the limits of chain-of-thought with multistep soft reasoning . In The Twelfth International Conference on Learning Representations

    2024

  16. [24]

    Zayne Rea Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2025. https://openreview.net/forum?id=w6nlcS8Kkn To cot or not to cot? chain-of-thought helps mainly on math and symbolic re...

    2025

  17. [25]

    Alon Talmor, Jonathan Herzig, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computat...

    2019 doi

  18. [26]

    Qwen Team. 2025. https://qwenlm.github.io/blog/qwq-32b/ QwQ-32B : Embracing the power of reinforcement learning

    2025

  19. [27]

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. https://openreview.net/forum?id=bzs4uPLXvi Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting . In Thirty-seventh Conference on Neural Information Proces...

    2023

  20. [28]

    Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasović, and Yonatan Belinkov. 2025. https://arxiv.org/abs/2502.14829 Measuring faithfulness of chains of thought by unlearning reasoning steps . Preprint, arXiv:2502.14829

    2025

  21. [29]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. https://openreview.net/forum?id=y10DM6R2r3 MMLU -pro: A more...

    2024

  22. [30]

    Zezhong Wang, Xingshan Zeng, Weiwen Liu, Yufei Wang, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2025. https://aclanthology.org/2025.findings-naacl.140/ Chain-of-probe: Examining the necessity and accuracy of C o T step-by-step . In Findings ...

    2025

  23. [31]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processi...

    2022

  24. [32]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  25. [33]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. https://arxiv.org/abs/2505.09388 Qwen3 technical report . Preprint, arXiv:2505.09388

    2025 arXiv

  26. [34]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. https://arxiv.org/abs/2405.15793 Swe-agent: Agent-computer interfaces enable automated software engineering . Preprint, arXiv:2405.15793

    2024 arXiv

  27. [35]

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.149 AGIE val: A human-centric benchmark for evaluating foundation models . In Findings of the Association for...

    2024 doi

  28. [36]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  29. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.