REVIEW 3 major objections 6 minor 2 cited by
MMATH: A Multilingual Benchmark for Mathematical Reasoning
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Reasoning in English while answering in the target language simultaneously improves accuracy and preserves language consistency, as shown on MMATH, a new 10-language mathematical reasoning benchmark.
desk verdict Useful benchmark, real off-target phenomenon, but the EN-Think training claim is confounded by machine-translation quality and needs a proper control before it can support the paper's central conclusion. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are MMATH itself and the EN-Think training configuration. MMATH is a set of 374 math problems from AIME, CNMO, and MATH-500 translated into Chinese, Arabic, Spanish, French, Japanese, Korean, Portuguese, Thai, and Vietnamese through a three-stage pipeline: LLM translation, cross-model iterative revision, and human verification. EN-Think is supervised fine-tuning on 3,000 examples where the question and final answer are in the target language but the chain-of-thought is in English. The language consistency ratio (LCR), computed with fastText language identification, is the metric that lets the paper quantify the off-target problem and verify that EN-Think preserves the answer language while switching the thinking language.
What would settle it
Have independent native-speaking mathematicians re-translate a random sample of MMATH problems from English and compare the two versions' difficulty; if the low-resource language versions such as Thai or Arabic come out systematically harder or more ambiguous than the English or Chinese versions, the per-language score gaps and the EN-Think advantage could be translation artifacts rather than reasoning effects.
Extended reading notes
Core claim
The paper's core discovery is that the language a model thinks in and the language it answers in can be decoupled to the benefit of both. On the MMATH benchmark, models that reason in English and answer in the target language outperform models that reason natively (66.72 vs 61.46 average) and models trained entirely in English (62.38), while achieving near-perfect answer-language consistency (97.61 LCR). The authors further observe that off-target thinking—thinking in a language other than the question's—is often more accurate than target-language thinking, especially for low-resource languages like Arabic and Thai, suggesting that English-centric internal reasoning is a strength to exploit rather than a failure to suppress.
Load-bearing premise
The benchmark translations preserve the exact mathematical semantics and difficulty of the original English problems, so cross-language score differences reflect model reasoning ability rather than translation artifacts.
Editorial extensions
If this is right
- Models trained or prompted to think in English and answer natively can serve monolingual users without an accuracy penalty; the paper shows that 3K training examples suffice for a 32B chat model to rival a dedicated reasoning model (66.72 vs 67.01).
- The off-target problem is measurable and can be mitigated by moderate thinking interventions such as discourse-initiated or question-restatement prompting, which raise answering LCR from below 60% to above 96%.
- Accuracy alone overstates a model's multilingual capability in low-resource languages, so evaluations should report answer-language consistency alongside answer correctness.
- Fine-tuning on English-only data (EN-SFT) actually worsens answer-language consistency (59.20 LCR), meaning the standard recipe for improving reasoning can actively hurt monolingual usability.
- Benchmark conclusions about multilingual readiness should separate thinking-language from answer-language, since even advanced models like DeepSeek-R1 show large per-language disparities on MMATH.
Reading between the lines
- If translation artifacts are controlled, the decoupling advantage may generalize to non-math reasoning, implying that English-centric internal scaffolding is a general property of these models rather than a math-specific trick.
- The EN-Think gain could partly reflect that English chain-of-thought traces distilled from DeepSeek-R1 are higher-quality than any currently available native-language traces; a testable extension would train Native-Think with equally strong native-language reasoning data.
- A natural next benchmark would add languages with more distant scripts, such as Hindi, Swahili, or Greek, to test whether the EN-Think recipe scales beyond the ten languages and whether answer-language consistency degrades with script distance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MMATH, a multilingual mathematical reasoning benchmark built from AIME 2024 (30 problems), AIME 2025 (15), CNMO (18), and a filtered subset of MATH-500 (311), translated into 10 languages (English, Chinese, Arabic, Spanish, French, Japanese, Korean, Portuguese, Thai, Vietnamese) via a three-stage pipeline (LLM translation, iterative LLM revision by three models, and human revision by certified native speakers). The authors evaluate chat and reasoning LLMs, introduce a language consistency ratio (LCR), document an off-target problem where models think or answer in unintended languages, and explore prompting (ATP, DIT, QRT) and three SFT strategies (EN-SFT, Native-Think, EN-Think). They report that EN-Think (English reasoning with native-language answers) achieves the best average accuracy (66.72) with high answering LCR (97.61), supporting the claim that reasoning in English while answering in the target language improves both performance and consistency.
Significance. If the central comparison is taken at face value, MMATH is a valuable community resource: it extends multilingual math evaluation from grade-school (MGSM) to competition-level problems, with a carefully documented translation and validation pipeline and public code and data. The empirical finding that EN-Think outperforms Native-Think is practically relevant for multilingual deployment. The paper also ships reproducible evaluation code, a transparent LCR metric validated by a 100-sample human check, and an honest limitations section. However, the load-bearing EN-Think versus Native-Think comparison is confounded by machine-translation quality of the training CoT, so the headline conclusion requires additional controlled experiments before it can be accepted as stated.
major comments (3)
- [Section 3.3.3 (Datasets Creation; Tables 7 and 8)] The claim that reasoning in English rather than in the native language drives the 5.26-point improvement is confounded by translation quality. All Light-R1 3K examples are originally in English; for Native-Think the entire example, including the long CoT, is machine-translated with GPT-4o-mini and never human-verified, whereas EN-Think uses the untranslated English CoT. Thus Native-Think may simply be trained on degraded targets. The paper validates benchmark translations (Section 2, with only 3% human edits) but reports no analogous quality check for the training-data translations. Please add a condition that holds CoT quality fixed across languages (e.g., human-validate the translated CoT, or back-translate the English CoT through the same pipeline as a control), or temper the causal claim.
- [Section 3.3.1 (Table 4 and subsection title)] The subsection title 'Off-target Thinking Increases Accuracy' makes a causal claim from observational data. The comparison conditions on the language the model happened to choose, which may correlate with problem difficulty, model confidence, or other latent factors. The DIT and QRT interventions in Section 3.3.2 alter more than just thinking language (they insert discourse markers or question restatements), so they do not isolate the effect of language. Please either reframe this as an associational observation or provide a controlled manipulation that varies only thinking language while holding reasoning content fixed.
- [Section 3.1 and all result tables] All results are point estimates averaged over 4 generations with no variance or significance testing. On small subsets such as AIME 2025 (15 problems, Table 13), a single-run difference of about 6.7 points is within sampling noise, so several per-language or small-gap comparisons (e.g., the 1-point ATP gains in Table 5, or per-language EN-Think versus EN-SFT gaps in Table 7) may not be robust. Please report per-run results, standard deviations, confidence intervals, or significance tests for the headline comparisons.
minor comments (6)
- [Throughout tables] The header 'A VG' should read 'AVG' in Tables 2, 5, 7, 13, and 14.
- [Abstract and Section 5] 'temporary reasoning models' should be 'contemporary reasoning models' in both places.
- [Figure 7] 'Buneo' is a typo for 'Bueno' (Spanish).
- [Table 11] 'Universidade NOV A de Lisboa' should be 'Universidade NOVA de Lisboa'.
- [Section 3.3.1] 'it may has already lost' should be 'it may have already lost'.
- [Section 3.3.2] The phrase 'more truthful multilingual ability' is vague; consider specifying what 'truthful' means operationally.
Circularity Check
No circularity: the benchmark is independent of the training data, and the EN-Think result is evaluated on held-out MMATH items.
full rationale
The paper's claims are empirical rather than derivational, and no equation-level or definition-level reduction of a result back to its own inputs can be exhibited. The MMATH benchmark is constructed independently of the fine-tuning experiments (Section 2): 374 English problems are translated through a three-stage GPT-4/Claude/Grok/human pipeline, and the models in Section 3 are then evaluated on this fixed benchmark using a held-out protocol (temperature 0.6, four repeated samples, macro-average). The training data in Section 3.3.3 come from Light-R1 3K, which is separate from MMATH; accuracy and LCR are measured on MMATH, not on the training distribution, so the EN-Think result (66.72, 97.61 answering LCR) is not an artifact of fitting to the test set. The high answering LCR of EN-Think is expected because the training examples explicitly contain native-language answers, but evaluating generalization on held-out native-language questions is a legitimate empirical test rather than a circular prediction. Similarly, DIT and QRT are simple prompt manipulations motivated by observed discourse patterns; they involve no fitted numeric parameters or test-score optimization, and their measured accuracy drops are empirical outcomes. The strongest potential concern, that Native-Think versus EN-Think confounds reasoning language with machine-translation quality of the CoT, is a real external-validity threat, and the Limitations section itself acknowledges that "synthesizing multilingual reasoning data remains a challenging problem" and that the "translation-based approach represents a preliminary attempt." But a confound is not circularity: the 5.26-point gap is not forced by construction, and no passage of the paper identifies the conclusion with an input or with a fitted parameter. There is no load-bearing self-citation chain; the only overlapping-author citation (Huang et al., 2023) is background related work, not the basis for the central claim. Therefore no circular step meets the evidentiary bar, and the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (5)
- domain assumption Translated MMATH questions preserve the mathematical semantics and difficulty of the English originals.
- domain assumption fastText language identification correctly classifies thinking and answer languages.
- domain assumption Answer extraction via OpenCompass's math extraction and math_verify is accurate.
- domain assumption The 374-problem benchmark, including only 15 AIME 2025 problems, is representative enough for reliable cross-language comparisons.
- domain assumption The Light-R1 3K subset is a suitable training distribution for multilingual complex reasoning after translation.
Cite this review
Pith. "Pith review of MMATH: A Multilingual Benchmark for Mathematical Reasoning." pith.science (2026). https://pith.science/paper/22IVXYKZ
@misc{pith2026250519126,
author = {Pith},
title = {Pith review of: MMATH: A Multilingual Benchmark for Mathematical Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/22IVXYKZ}},
note = {Machine review of arXiv:2505.19126}
}
read the original abstract
The advent of large reasoning models, such as OpenAI o1 and DeepSeek R1, has significantly advanced complex reasoning tasks. However, their capabilities in multilingual complex reasoning remain underexplored, with existing efforts largely focused on simpler tasks like MGSM. To address this gap, we introduce MMATH, a benchmark for multilingual complex reasoning spanning 374 high-quality math problems across 10 typologically diverse languages. Using MMATH, we observe that even advanced models like DeepSeek R1 exhibit substantial performance disparities across languages and suffer from a critical off-target issue-generating responses in unintended languages. To address this, we explore strategies including prompting and training, demonstrating that reasoning in English and answering in target languages can simultaneously enhance performance and preserve target-language consistency. Our findings offer new insights and practical strategies for advancing the multilingual reasoning capabilities of large language models. Our code and data could be found at https://github.com/RUCAIBox/MMATH.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 2 Pith papers
-
Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAG
Teacher-regularized on-policy RL with a reverse-KL anchor on student prefixes improves language consistency and evidence-grounded correctness for English-evidence cross-lingual RAG, acting as a safety net against larg...
-
SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
SPEED-Bench is a new standardized benchmark for speculative decoding that supplies semantically diverse qualitative data and throughput-oriented splits across concurrency levels, integrated with vLLM and TensorRT-LLM.
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
arXiv 2023
-
[4]
Liang Chen, Shuming Ma, Dongdong Zhang, Furu Wei, and Baobao Chang. 2023 a . https://doi.org/10.18653/v1/2023.findings-acl.608 On the off-target problem of zero-shot multilingual neural machine translation . In Findings of the Association for Computational Linguistics: ACL 2023, pages 9542--9558, Toronto, Canada. Association for Computational Linguistics
-
[5]
Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. 2023 b . Breaking language barriers in multilingual mathematical reasoning: Insights and observations. arXiv preprint arXiv:2310.20246
arXiv 2023
-
[6]
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168
arXiv 2021
-
[7]
Marta R Costa-Juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672
arXiv 2022
-
[8]
Hugging Face. 2025. https://github.com/huggingface/open-r1 Open r1: A fully open reproduction of deepseek-r1
2025
Show all 34 references
-
[9]
Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2023. Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179
2023 arXiv
-
[10]
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519
2025 arXiv
-
[11]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948
2025 arXiv
-
[12]
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung-Yeung Shum Xiangyu Zhang. 2025. Open-reasoner-zero: An open source approach to scaling reinforcement learning on the base model. https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero
2025
-
[13]
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. arXiv preprint arXiv:2305.07004
2023 arXiv
-
[14]
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720
2024 arXiv
-
[15]
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759
2016 arXiv
-
[16]
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. In The Twelfth International Conference on Learning Representations
2023
-
[17]
Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. 2024. Is translation all you need? a study on solving multilingual tasks with large language models. arXiv preprint arXiv:2403.10258
2024 arXiv
-
[18]
Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. 2025. Reasoning models can be effective without thinking. arXiv preprint arXiv:2504.09858
2025 arXiv
-
[19]
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. http://arxiv.org/abs/2501.19393 s1: Simple test-time scaling
2025 arXiv
-
[20]
OpenCompass. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass
2023
-
[21]
Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2024. Mutual reasoning makes smaller llms stronger problem-solvers. arXiv preprint arXiv:2408.06195
2024 arXiv
-
[22]
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057
2022 arXiv
-
[23]
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314
2024 arXiv
-
[24]
Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne. 2025. Linguistic generalizability of test-time scaling in mathematical reasoning. arXiv preprint arXiv:2502.17407
2025 arXiv
-
[25]
Gemma Team. 2025 a . https://goo.gle/Gemma3Report Gemma 3
2025
-
[26]
Qwen Team. 2025 b . https://qwenlm.github.io/blog/qwq-32b/ Qwq-32b: Embracing the power of reinforcement learning
2025
-
[27]
Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, et al. 2025. Polymath: Evaluating mathematical reasoning in multilingual contexts. arXiv preprint arXiv:2504.18428
2025
-
[28]
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. 2025. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:...
2025 arXiv
-
[29]
Tong Wu, Chong Xiang, Jiachen T Wang, and Prateek Mittal. 2025. Effectively controlling reasoning models through thinking intervention. arXiv preprint arXiv:2503.24370
2025 arXiv
-
[30]
Arinjay Wyawhare. 2023. Comparative analysis of multilingual text classification & identification through deep learning and embedding visualization. arXiv preprint arXiv:2312.03789
2023 arXiv
-
[31]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...
2024 arXiv
-
[32]
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. http://arxiv.org/abs/2502.03387 Limo: Less is more for reasoning
2025 arXiv
-
[33]
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. 2024. Free process rewards without process labels. arXiv preprint arXiv:2412.01981
2024 arXiv
-
[34]
Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. 2024. https://doi.org/10.18653/v1/2024.findings-acl.498 Question translation training for better multilingual reasoning . In Findings of the Association for Computational Linguistics: ACL 2024...
2024 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.