REVIEW 13 cited by
GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks. However, there are increasing debates regarding whether these models truly understand and apply mathematical knowledge or merely rely on shortcuts for mathematical reasoning. One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly. This motivates us to evaluate the robustness of LLMs' math reasoning capability by testing a wide range of question variations. We introduce the adversarial grade school math (GSM-Plus) dataset, an extension of GSM8K augmented with various mathematical perturbations. Our experiments on 25 LLMs and 4 prompting techniques show that while LLMs exhibit different levels of math reasoning abilities, their performances are far from robust. In particular, even for problems that have been solved in GSM8K, LLMs can make mistakes when new statements are added or the question targets are altered. We also explore whether more robust performance can be achieved by composing existing prompting methods, in which we try an iterative method that generates and verifies each intermediate thought based on its reasoning goal and calculation result.
Forward citations
Cited by 13 Pith papers
-
Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation
A disentangled evaluation of math word problems finds arithmetic computation, not abstract formulation, is the main bottleneck for LLMs without chain-of-thought, and chain-of-thought mainly helps computation.
-
Are Arithmetic Heuristic Neurons Form-Invariant? A Mechanistic Analysis of Symbols, Text, and Code in LLMs
Llama-3 arithmetic is computed by a shared neuron set across symbolic, text, and code formats, and cross-format failures are activation-state differences, not distinct circuits.
-
Implicit Reasoning Steering via Concept Chaining
Reinforcement-learning-optimized concept-chain paragraphs covertly steer language-model multiple-choice preferences after continued pretraining, with far lower detectability than direct paraphrases.
-
Don't Commit Alone: Joint Token Commitment in Diffusion Large Language Models
Marker-gated re-application of a dLLM’s last layers lets a commit bundle coordinate before greedy write, improving accuracy on all six evaluated benchmarks under matched greedy inference.
-
Model soups need only one ingredient
A single checkpoint, edited by splitting each layer's update with SVD and reweighting the high- and low-energy parts, reaches soup-level OOD robustness without multi-model training.
-
FLAMES: Improving LLM Math Reasoning via a Fine-Grained Analysis of the Data Synthesis Pipeline
A controlled study of synthetic math data generation yields a new dataset blend (FLAMES) that improves fine-tuned 7B model scores on MATH and OlympiadBench, with caveats on checkpoint selection.
-
TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewards
TRPrompt trains an 8B prompt model directly on natural-language textual rewards and reports the highest accuracies on GSMHard and MATH among the compared methods.
-
CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective
When math problems are probed through nine human-cognitive dimensions, seven leading LLMs' effective pass rates drop 30 to 40 points below standard accuracy.
-
RTTC: Reward-Guided Collaborative Test-Time Compute
RTTC uses a reward model to route each query to direct inference, retrieval-augmented generation, or test-time training, plus caching to cut redundant work.
-
Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding
DeepSeek-R1-distilled models show higher multi-document QA accuracy than their base counterparts and flatter position-bias curves, especially with 50-80 documents.
-
GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models
The paper releases GSM-Plus-BN, a human-verified Bengali translation of the GSM-Plus perturbed math benchmark, and reports accuracy baselines for six open LLMs under standard and CoT prompting.
-
Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation
AdvDistill uses group relative advantages computed from rule-based rewards to weight teacher responses during distillation, reportedly improving a 1.5B student on math tasks beyond its 7B teacher.
-
LFTF: Locating First and Then Fine-Tuning for Mitigating Gender Bias in Large Language Models
A block-localizing fine-tuning method for gender debiasing is presented, but its stated loss is inconsistent with its reported behavior and the evaluation tables contain duplicate rows.
Discussion (0). Sign in to comment.