REVIEW 9 cited by
Common 7B Language Models Already Possess Strong Math Capabilities
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Mathematical capabilities were previously believed to emerge in common language models only at a very large scale or require extensive math-related pre-training. This paper shows that the LLaMA-2 7B model with common pre-training already exhibits strong mathematical abilities, as evidenced by its impressive accuracy of 97.7% and 72.0% on the GSM8K and MATH benchmarks, respectively, when selecting the best response from 256 random generations. The primary issue with the current base model is the difficulty in consistently eliciting its inherent mathematical capabilities. Notably, the accuracy for the first answer drops to 49.5% and 7.9% on the GSM8K and MATH benchmarks, respectively. We find that simply scaling up the SFT data can significantly enhance the reliability of generating correct answers. However, the potential for extensive scaling is constrained by the scarcity of publicly available math questions. To overcome this limitation, we employ synthetic data, which proves to be nearly as effective as real data and shows no clear saturation when scaled up to approximately one million samples. This straightforward approach achieves an accuracy of 82.6% on GSM8K and 40.6% on MATH using LLaMA-2 7B models, surpassing previous models by 14.2% and 20.8%, respectively. We also provide insights into scaling behaviors across different reasoning complexities and error types.
Forward citations
Cited by 9 Pith papers
-
Mitigating Object Hallucination via Robust Local Perception Search
A training-free decoding method that uses an MLLM's own local object descriptions as a reward prior, combined with CLIP similarity, to cut object hallucination, especially under adversarial image noise.
-
Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment
Post-training alignment on synthetic graph data yields statistically significant gains on some real-world implicit-graph tasks, but gains are inconsistent and multi-step reasoning remains fragile.
-
How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning
On three controlled tasks, the optimal number of backtracks in SFT warm-up data grows with task difficulty, while trajectory correctness has little effect on final RL performance.
-
Multimodal Behavioral Patterns Analysis with Eye-Tracking and LLM-Based Reasoning
A human-AI framework uses horizontal and vertical segmentation with LLMs, expert co-scoring, and LSTM anomaly detection to extract behavioral patterns from eye-tracking data.
-
We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems
MCP-powered LLM agents are vulnerable to prompt injection from third-party services, and simple detection or filtering defenses do not reliably stop these attacks.
-
SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
SwS uses failures during RL training to synthesize targeted math problems, improving reasoning accuracy on eight benchmarks.
-
Paths to Causality: Finding Informative Subgraphs Within Knowledge Graphs for Knowledge-Based Causal Discovery
A learning-to-rank model chooses informative knowledge-graph paths between entity pairs, and adding the top path to zero-shot prompts improves LLM causal classification by up to 44.4 F1 points.
-
Two Experts Are All You Need for Steering Thinking: Reinforcing Cognitive Effort in MoE Reasoning Models Without Additional Training
Reinforcing the two experts most correlated with thinking tokens improves reasoning accuracy and efficiency in MoE large reasoning models, with gains of up to 10 points on AIME benchmarks.
-
A Survey on Large Language Models for Mathematical Reasoning
Recent advances in LLM mathematical reasoning are organized into comprehension and generation phases, covering methods from prompting to test-time scaling.
Discussion (0). Continue with ORCID to comment.