REVIEW 9 cited by
Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The past decade has witnessed dramatic gains in natural language processing and an unprecedented scaling of large language models. These developments have been accelerated by the advent of few-shot techniques such as chain of thought (CoT) prompting. Specifically, CoT pushes the performance of large language models in a few-shot setup by augmenting the prompts with intermediate steps. Despite impressive results across various tasks, the reasons behind their success have not been explored. This work uses counterfactual prompting to develop a deeper understanding of CoT-based few-shot prompting mechanisms in large language models. We first systematically identify and define the key components of a prompt: symbols, patterns, and text. Then, we devise and conduct an exhaustive set of experiments across four different tasks, by querying the model with counterfactual prompts where only one of these components is altered. Our experiments across three models (PaLM, GPT-3, and CODEX) reveal several surprising findings and brings into question the conventional wisdom around few-shot prompting. First, the presence of factual patterns in a prompt is practically immaterial to the success of CoT. Second, our results conclude that the primary role of intermediate steps may not be to facilitate learning how to solve a task. The intermediate steps are rather a beacon for the model to realize what symbols to replicate in the output to form a factual answer. Further, text imbues patterns with commonsense knowledge and meaning. Our empirical and qualitative analysis reveals that a symbiotic relationship between text and patterns explains the success of few-shot prompting: text helps extract commonsense from the question to help patterns, and patterns enforce task understanding and direct text generation.
Forward citations
Cited by 9 Pith papers
-
What Does Chain-of-Thought Contribute at Probe Time? Evidence for Local Co-Occurrence Activation
CoT probe-time gains arise primarily from lexical activation and short-range token co-occurrence rather than sentence-level logical derivation.
-
From Noise to Diversity: Random Embedding Injection in LLM Reasoning
Random Soft Prompts (RSPs) sampled from the embedding distribution improve Pass@N on reasoning benchmarks by increasing early-stage token diversity without any training.
-
UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp
A unified KG-plus-live-web data pipeline covering all three multimodal BrowseComp information-flow patterns, plus an exploration-degree filter, yields a 35B agent at 54.4 avg accuracy.
-
Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
LLM attention maps reveal a preplan-and-anchor pattern, and reweighting RL credit toward the flagged tokens improves math/QA reasoning.
-
APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuning
A profiling-guided, LLM-driven framework combines structured pruning and mixed-precision quantization-aware training, reporting 13-18x bit-operation reductions with modest accuracy loss on ImageNet and CIFAR-10.
-
Rethinking the Chain-of-Thought: The Roles of In-Context Learning and Pre-trained Priors
The paper finds that as exemplar count increases, LLMs shift from pretrained priors to in-context signals, and long chain-of-thought prompts induce longer reasoning chains that often improve accuracy.
-
Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion
LLM hidden states encode enough information to predict chain-of-thought success before any reasoning tokens are generated, outperforming a text-only classifier.
-
Investigating the Effects of Cognitive Biases in Prompts on Large Language Model Outputs
Injecting explicit suggestions or biased recollections into prompts reduces LLM accuracy on multiple-choice QA tasks, and attention weights shift toward the suggested answer.
-
SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
SCOUT combines progressive distillation with a cross-attention module to make recursive latent reasoning work through fine-tuning, yielding up to 1.8% accuracy gains over standard fine-tuning.
Discussion (0). Continue with ORCID to comment.