REVIEW 9 cited by
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs). CoT explicitly encourages the LLM to generate intermediate rationales for solving a problem, by providing a series of reasoning steps in the demonstrations. Despite its success, there is still little understanding of what makes CoT prompting effective and which aspects of the demonstrated reasoning steps contribute to its performance. In this paper, we show that CoT reasoning is possible even with invalid demonstrations - prompting with invalid reasoning steps can achieve over 80-90% of the performance obtained using CoT under various metrics, while still generating coherent lines of reasoning during inference. Further experiments show that other aspects of the rationales, such as being relevant to the query and correctly ordering the reasoning steps, are much more important for effective CoT reasoning. Overall, these findings both deepen our understanding of CoT prompting, and open up new questions regarding LLMs' capability to learn to reason in context.
Forward citations
Cited by 9 Pith papers
-
Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
LLM attention maps reveal a preplan-and-anchor pattern, and reweighting RL credit toward the flagged tokens improves math/QA reasoning.
-
Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting
Standardized modular threat-hunting workflows (CyberTeam) improve LLM performance on blue team tasks compared to open-ended ICL, CoT, and ToT prompting across 30 tasks and 452k samples.
-
WebGuard: Building a Generalizable Guardrail for Web Agents
WebGuard introduces an action-level risk dataset for web agents and shows that a fine-tuned 7B model improves risk-prediction accuracy from about 38% to 80% and high-risk recall from 20% to 76%, still below deployment...
-
Distinct Computations Emerge From Compositional Curricula in In-Context Learning
When transformer models see easy component examples before a harder combined math problem in one prompt, they solve unseen versions of the combined problem and store intermediate steps internally, unlike models traine...
-
Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
The monotonicity of token probabilities during initial decoding predicts chain-of-thought gains, enabling dynamic selection between CoT and direct answers.
-
SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning
SuperRL switches per-instance between policy-gradient updates and supervised fine-tuning, using the presence of any nonzero reward as the switch signal.
-
Toward Structured Knowledge Reasoning: Contrastive Retrieval-Augmented Generation on Experience
CoRE improves structured knowledge reasoning by retrieving both correct and incorrect past examples into the prompt, using MCTS-generated experience memory.
-
A Statistical Physics of Language Model Reasoning
A switching linear dynamical system on a 40-dimensional projection of LLM hidden states captures about half the variance of reasoning trajectories and predicts belief shifts during adversarial prompts.
-
Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects
A survey that classifies chain-of-thought methods for autonomous driving into modular, logical, and reflective pipelines, and proposes three evolutionary stages from direct prompting to reinforcement learning.
Discussion (0). Continue with ORCID to comment.