REVIEW 7 cited by
HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce HARDMath, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on HARDMath-mini, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few-shot Chain-of-Thought prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct a detailed error analysis to gain insights into the failure cases of LLMs. These results demonstrate limitations of current LLM performance on advanced graduate-level applied math problems and underscore the importance of datasets like HARDMath to advance mathematical abilities of LLMs.
Forward citations
Cited by 7 Pith papers
-
AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
A 245-problem advanced proof benchmark plus 888 expert-labeled trajectories shows frontier LLMs remain far from reliable advanced proof generation and verification.
-
SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy
A new benchmark of 2,703 automatically generated multimodal questions for scanning probe microscopy, plus a modified F1 metric that penalizes over-selection and labels model 'personalities'.
-
EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning
EngTrace, a 1,350-instance symbolic engineering benchmark with gold reasoning traces, shows frontier LLMs outperform math-specialized small models and that trace verification reveals a complexity cliff.
-
CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization
A critic model trained with reinforcement learning judges semantic correctness of Lean 4 formalizations, and using it as a filter sharply improves autoformalization accuracy.
-
Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering
RAPL combines LLM-rationalized path labels, line graph transformation, and path-based decoding to improve graph retrieval for KGQA, reporting state-of-the-art results on WebQSP and CWQ.
-
Power Law Guided Dynamic Sifting for Efficient Attention
SiftAttention skips top-k sorting in sparse attention by thresholding attention weights with a threshold predicted from a power-law fit of score quantiles over generation steps.
-
ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning
Distributed rollout workers under a bounded-staleness budget can keep a centralized learner saturated and cut LLM post-training cost by roughly a third at matched reward.
Discussion (0). Continue with ORCID to comment.