REVIEW 7 cited by
Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Mathematical reasoning tasks pose significant challenges for large language models (LLMs) because they require precise logical deduction and sequence analysis. In this work, we introduce the concept of critical tokens -- elements within reasoning trajectories that significantly influence incorrect outcomes. We present a novel framework for identifying these tokens through rollout sampling and demonstrate their substantial divergence from traditional error tokens. Through extensive experiments on datasets such as GSM8K and MATH500, we show that identifying and replacing critical tokens significantly improves model accuracy. We propose an efficient methodology for pinpointing these tokens in large-scale datasets using contrastive estimation and extend this framework to enhance model training processes with direct preference optimization (DPO). Experimental results on GSM8K and MATH500 benchmarks with the widely used models Llama-3 (8B and 70B) and Deepseek-math (7B) demonstrate the effectiveness of the proposed approach, cDPO. Our results underscore the potential of leveraging critical tokens to reduce errors in reasoning tasks, advancing the development of AI systems capable of robust logical deduction. Our code, annotated datasets, and trained models are available at https://github.com/chenzhiling9954/Critical-Tokens-Matter to support and encourage future research in this promising field.
Forward citations
Cited by 7 Pith papers
-
Rethinking On-Policy Self-Distillation for Thinking Models
Privileged-context on-policy self-distillation degrades thinking models' long-budget accuracy by suppressing forking and self-correction behaviors, while helping instruction-tuned models.
-
Embarrassingly Simple Self-Distillation Improves Code Generation
Simple self-distillation—fine-tuning a code model on its own temperature-sampled, truncated outputs—raises LiveCodeBench pass@1 substantially without verifiers, teachers, or RL.
-
Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
LLM attention maps reveal a preplan-and-anchor pattern, and reweighting RL credit toward the flagged tokens improves math/QA reasoning.
-
Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
Reasoning tokens like 'Hmm' and 'Wait' mark steps where a model's internal state carries unusually high dependence with the correct answer, and suppressing them hurts accuracy.
-
Probability-Consistent Preference Optimization for Enhanced LLM Reasoning
PCPO selects preference pairs by combining correct-answer status with token-level probability consistency, then trains with a weighted DPO+NLL loss, yielding small and partly inconsistent gains over outcome-only metho...
-
RAST: Reasoning Activation in LLMs via Small-model Transfer
Injecting the logit gap between a small RL-trained model and its base into a larger base model at decoding time improves reasoning accuracy on math and code benchmarks, sometimes matching or exceeding RL training of t...
-
Reinforcing Video Reasoning with Focused Thinking
A GRPO variant with token-level KL weighting and partial-credit rewards improves video-QA on modified multi-answer benchmarks, but transfer to original single-answer benchmarks is not established.
Discussion (0). Sign in to comment.