What Makes Math Word Problems Challenging for LLMs?

Ekaterina Kochmar; KV Aditya Srivatsa

What Makes Math Word Problems Challenging for LLMs?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.11369 v2 pith:V6XYYSEH submitted 2024-03-17 cs.CL

KV Aditya Srivatsa , Ekaterina Kochmar This is my paper

classification cs.CL

keywords llmsmwpschallengingmakesmathproblemswhatword

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

This paper investigates the question of what makes math word problems (MWPs) in English challenging for large language models (LLMs). We conduct an in-depth analysis of the key linguistic and mathematical characteristics of MWPs. In addition, we train feature-based classifiers to better understand the impact of each feature on the overall difficulty of MWPs for prominent LLMs and investigate whether this helps predict how well LLMs fare against specific categories of MWPs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Exploring Task Performance with Interpretable Models via Sparse Auto-Encoders
cs.CL 2025-07 conditional novelty 6.0 of 10

Sparse-autoencoder features from LLMs trigger automatic prompt reformulation, yielding consistent gains on mathematical reasoning and metaphor detection.

Pith tools