Pith

open record

sign in

arxiv: 2607.05992 · v1 · pith:W7GXBKE7 · submitted 2026-07-07 · cs.CL · cs.AI

PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages

Daryna Dementieva , Nikolay Babakov , Kathy H\"ammerl , Ilseyar Alimova , Jind\v{r}ich Libovick\'y , Shu Okabe , Miras Baisbay , Lukas Edman
show 9 more authors
Abrorkhon Inomkhujaev Antonia Karamolegkou Mateusz Lango Volkan \"Ozer Nikola Selic Subhankar Swain Tsedeniya Kinfe Temesgen Galit Bary Weisberg Alexander Fraser
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved 2026-07-08 19:05 UTCgrok-4.5pith:W7GXBKE7record.jsonopen to challenge →

Figure 1
Figure 1. Figure 1: Plura—from Latin, “more”: our extension of PolyMath (Wang et al., 2025) 4-level mathemati￾cal reasoning benchmark to 18 underrepresented lan￾guages through comprehensive human annotation of pre-computed translations. et al., 2021), AIME, MathArena (Balunovic et al., 2025), OMEGA (Sun et a… reproduced from arXiv: 2607.05992
classification cs.CL cs.AI
keywords multilingual mathematical reasoningunderrepresented languagesLLM evaluationPolyMath extensioninstruction followinglow-resource languagesbenchmark constructionreasoning LLMs
0
0 comments X

The pith

PluraMath extends math-reasoning tests to 18 underrepresented languages and finds a persistent high-resource gap across 27 models

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Mathematical reasoning is a standard way to test and tune large language models, but almost every public benchmark is still dominated by English, Chinese, and a few other high-resource languages. This paper introduces PluraMath, a carefully validated extension of the existing PolyMath suite that adds 18 more languages spanning six families, from mid-resource to extremely low-resource settings. Native speakers checked pre-computed translations so that problem content, difficulty, and answer format stay equivalent. The authors then run 27 reasoning models at four scales on the new set and show that scores remain systematically lower on the underrepresented languages. Better multilingual instruction-following, not raw scale alone, is the main factor that closes the gap. By open-sourcing the data, the translation pipeline, and the evaluation code, the work aims to make it easier for underrepresented language communities to build their own math-reasoning benchmarks.

Core claim

Across 27 reasoning LLMs spanning small, mid-size, large, and closed-source ensembles, mathematical reasoning accuracy on PluraMath remains markedly lower for the 18 newly added underrepresented languages than for high-resource ones; stronger results track mainly with a model’s ability to follow instructions in those languages rather than with parameter count alone.

What carries the argument

A human-curated translation-and-validation pipeline that takes PolyMath problems, produces candidate translations, and has native speakers verify mathematical equivalence, difficulty, and answer format across 18 additional languages spanning six families.

If this is right

  • Benchmarking suites that ignore mid- and low-resource languages will systematically overstate the multilingual reasoning ability of current LLMs.
  • Improvements in multilingual instruction-following should raise math-reasoning scores on underrepresented languages more reliably than simply scaling model size.
  • Open release of the validated items and pipeline lowers the cost for other language communities to build parallel math-reasoning tests.
  • Closed-source ensembles that already show stronger instruction following are likely to retain their relative advantage on the new languages unless open models close the instruction gap.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If instruction-following is the dominant bottleneck, targeted multilingual instruction tuning on math templates may close more of the gap than additional pre-training on raw low-resource text.
  • The same validation pipeline could be reused for other structured reasoning domains (code, formal logic, scientific word problems) where surface form must not alter the underlying answer.
  • Persistent gaps on extreme low-resource languages may eventually force model developers to treat those languages as first-class evaluation axes rather than optional extras.

Load-bearing premise

Native-speaker checks of pre-computed translations are assumed to keep problem meaning, difficulty, and answer format equivalent enough that score gaps can be blamed on model reasoning rather than leftover translation or template artifacts.

What would settle it

Re-translate a stratified sample of PluraMath items with an independent human pipeline (or back-translate and re-solve) and re-evaluate the same 27 models; if the high-versus-low-resource gap shrinks or vanishes while English scores stay stable, the original gap was an artifact of residual translation inequivalence.

Figures

Figures reproduced from arXiv: 2607.05992 by Abrorkhon Inomkhujaev, Alexander Fraser, Antonia Karamolegkou, Daryna Dementieva, Galit Bary Weisberg, Ilseyar Alimova, Jind\v{r}ich Libovick\'y, Kathy H\"ammerl, Lukas Edman, Mateusz Lango, Miras Baisbay, Nikola Selic, Nikolay Babakov, Shu Okabe, Subhankar Swain, Tsedeniya Kinfe Temesgen, Volkan \"Ozer.

Figure 2
Figure 2. Figure 2: Comparison of math task lengths across high-resource languages from PolyMath (English and Russian) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Math answer correctness distributions per language (a) and per model (b). We compare high-resource and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (A): Correlations with translations capabilities analysis. (B): Human reasoning assessment results. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples of a translated math problem from [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Examples of a translated math problem from [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Examples of a translated math problem from [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Examples of a translated math problem from [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Fuul box-plot comparison of math task sample lengths between four high-resource languages from [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The translated per language final instructions which were added in the end of main tasks. English [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Aggregated DW=ACC base prompting scores distributions per all languages across all models and all levels.We observe a clear trend in mean scores that correlates with language resource rankings, with severely underrepresented languages exhibiting low performance even on low-difficulty tasks [PITH_FULL_IMAGE:figures/full_fig_p031_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Answers correctness scores distributions for [PITH_FULL_IMAGE:figures/full_fig_p032_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Distribution of reasoning and answers length for [PITH_FULL_IMAGE:figures/full_fig_p033_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Distribution of models’ reasoning and answers length for [PITH_FULL_IMAGE:figures/full_fig_p034_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Interface used for the human assessment of models’ answers and reasoning part. [PITH_FULL_IMAGE:figures/full_fig_p037_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Example of a correct answer and reasoning from [PITH_FULL_IMAGE:figures/full_fig_p039_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Example of a correct reasoning and semantically correct but mismatched answer from [PITH_FULL_IMAGE:figures/full_fig_p039_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: DeepSeek-V3.2 typically reasoned in the target language, but quite often did not finish correct answer generation due to repetitive self-checking loops. Nemotron3_nano_omni_30ba3b frequently produced non-fluent reasoning traces ( [PITH_FULL_IMAGE:figures/full_fig_p040_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Example of a mixed languages reasoning and not so fluent answer from [PITH_FULL_IMAGE:figures/full_fig_p040_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Nemotron3_nano_omni_30ba3b could also produce absolutely non-fluent reasoning with even leaked control tokens. behavior, alternating between target-language and primarily English reasoning. Most incorrect predictions resulted from incomplete reasoning traces as in [PITH_FULL_IMAGE:figures/full_fig_p041_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Example of the unfinished reasoning from [PITH_FULL_IMAGE:figures/full_fig_p042_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Gemma-3-4b consistently reasoned fluently and in the target language, but there could have been wrong reasoning steps [PITH_FULL_IMAGE:figures/full_fig_p043_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Translation with reasoning models example for the Ukrainian sample. Some models tend to hallucinate [PITH_FULL_IMAGE:figures/full_fig_p045_23.png] view at source ↗
read the original abstract

Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites. The recently released PolyMath (Wang et al., 2025) dataset represents a significant step forward, yet its coverage is still limited to 18 only high-resource languages. To address this gap, we introduce PluraMath, an extension of PolyMath to 18 additional {underrepresented languages spanning 6 language families -- ranging from mid-resource to extreme low-resource settings. We constructed the dataset through a human-curated pipeline, where native speakers thoroughly validated pre-computed translations. Using PluraMath, we then benchmark 27 reasoning LLMs across four model scales -- small, mid-size, large, and closed-source ensembles -- probing the multilingual mathematical reasoning capabilities of state-of-the-art models under diverse linguistic conditions. Our fine-grained analysis confirms a persistent gap in mathematical reasoning performance between high-resource and underrepresented languages, with stronger results largely associated with better instruction-following ability. We fully open-source our dataset, data acquisition pipeline, and evaluation framework, with the goal of lowering the barrier to multilingual benchmark development for underrepresented communities.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 6 minor

Summary. The manuscript introduces PluraMath, an extension of the PolyMath mathematical-reasoning benchmark from 18 high-resource languages to 18 additional underrepresented languages spanning six language families (mid- to extreme low-resource). Items are produced via a human-curated pipeline in which native speakers validate pre-computed translations. The authors then evaluate 27 reasoning LLMs across four scale bands (small, mid-size, large, and closed-source ensembles) and report a persistent performance gap between high-resource and underrepresented languages, with stronger scores largely associated with better instruction-following. The dataset, acquisition pipeline, and evaluation framework are fully open-sourced.

Significance. If the items preserve mathematical content, difficulty, and answer format across languages, PluraMath would be a concrete and timely contribution: it expands multilingual math evaluation into languages that existing suites systematically omit, and the multi-scale evaluation of 27 models supplies a useful snapshot of current capabilities. Full open-sourcing of the dataset, pipeline, and evaluation framework is a clear strength that can lower the barrier for community-driven extension to further underrepresented languages. The secondary link between performance and instruction-following is a potentially actionable finding for model development, provided it is not confounded by evaluation artifacts.

major comments (3)
  1. The central claim—that measured high- vs underrepresented-language gaps reflect multilingual mathematical reasoning rather than evaluation artifacts—depends on solution-preserving equivalence of PluraMath items to their PolyMath sources. The abstract and construction description state that native speakers “thoroughly validated” pre-computed translations, but do not report residual error rates, inter-annotator agreement, or an explicit validation protocol that requires re-solving for a unique correct answer under a fixed rubric (as opposed to surface fluency or grammaticality). Without those controls, residual wording ambiguity, difficulty shift, script/format drift, or answer-format mismatch can systematically depress underrepresented-language scores and inflate the apparent reasoning gap. This is load-bearing for attribution; the manuscript should quantify validation quality and, where
  2. The secondary finding that stronger results are “largely associated with better instruction-following ability” is vulnerable to the same confound if instruction-following is scored on the same translated prompts or templates. The manuscript should (i) define how instruction-following is measured (separate probe vs. same PluraMath items), (ii) state whether instruction templates were language-adapted and how format consistency was enforced, and (iii) show that the association survives controls for residual translation quality or template mismatch. Otherwise the association cannot be cleanly attributed to model capability rather than prompt surface form.
  3. Cross-language and cross-scale gap claims need explicit statistical support and matched evaluation conditions. The fine-grained analysis should report per-language and per-family scores with uncertainty (e.g., bootstrap CIs or paired tests against the PolyMath high-resource baseline under identical decoding and answer-extraction settings), and should document any language-specific answer-normalization rules. Without this, “persistent gap” remains a qualitative summary rather than a secured empirical result, especially for extreme low-resource languages where small absolute score differences can be dominated by extraction failures.
minor comments (6)
  1. Clarify the exact relationship to PolyMath (Wang et al., 2025): which problem subsets were reused, whether difficulty tiers were preserved, and whether any items were rewritten rather than translated.
  2. List the 18 underrepresented languages and their language-family groupings early (table or figure) so readers can map resource level to results without hunting the appendix.
  3. Define the four model-scale bands (parameter ranges or explicit model lists) in the main text so the “27 models across four scales” claim is auditable without supplementary material.
  4. Report decoding hyperparameters, answer-extraction method, and any language-specific post-processing in a single reproducible evaluation subsection; open-sourcing the framework is valuable but the paper should still document the protocol.
  5. In the abstract and introduction, avoid implying that PolyMath covers “only high-resource languages” without a brief resource-level criterion (e.g., Common Crawl share or labeled-data availability) so the high- vs underrepresented contrast is operationally clear.
  6. If figures plot high-resource vs underrepresented aggregates, add per-language strip or family-level breakdowns so outliers in extreme low-resource settings are visible.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The three major comments correctly identify load-bearing reporting gaps around (i) translation/validation quality controls, (ii) the operational definition of instruction-following and possible confounds with prompt surface form, and (iii) statistical support for cross-language and cross-scale gap claims. We agree that these points must be addressed before the central attribution—that measured gaps reflect multilingual mathematical reasoning rather than evaluation artifacts—can be secured. We will revise the manuscript accordingly: expand the validation protocol with residual-error and agreement metrics, clarify how instruction-following is measured and controlled, and add per-language/family scores with uncertainty under matched evaluation conditions. Full open-sourcing of dataset, pipeline, and evaluation code remains unchanged and will incorporate the additional diagnostics. We believe these revisions directly answer the referee’s concerns and strengthen the contribution.

read point-by-point responses

  1. Referee: The central claim—that high- vs underrepresented-language gaps reflect multilingual mathematical reasoning rather than evaluation artifacts—depends on solution-preserving equivalence of PluraMath items to PolyMath sources. The manuscript states that native speakers “thoroughly validated” pre-computed translations, but does not report residual error rates, inter-annotator agreement, or an explicit validation protocol that requires re-solving for a unique correct answer under a fixed rubric (vs. surface fluency). Without those controls, residual ambiguity, difficulty shift, script/format drift, or answer-format mismatch can systematically depress underrepresented-language scores and inflate the apparent reasoning gap. The manuscript should quantify validation quality.

    Authors: We agree that solution-preserving equivalence is load-bearing for attribution, and that the current manuscript under-specifies validation quality. Our pipeline already required native-speaker validators to check that each item retained the same mathematical content, difficulty intent, and unique gold answer as the PolyMath source (not merely fluency), with rejection and re-translation on failure. However, we did not report residual error rates, inter-annotator agreement, or the full rubric in the text. In revision we will: (1) spell out the explicit validation protocol and acceptance criteria (including re-solving / answer uniqueness under a fixed rubric); (2) report residual error rates from a held-out double-annotation sample, stratified by language and resource band; (3) report inter-annotator agreement on accept/reject and on answer equivalence; and (4) document any residual issues (script/format drift, answer-format mismatch) and how they were handled. Where residual risk remains for extreme low-resource languages, we will state it explicitly rather than over-claim equivalence. These additions will appear in the dataset-construction section and appendix, with the open-sourced pipeline updated to match. revision: yes

  2. Referee: The secondary finding that stronger results are “largely associated with better instruction-following ability” is vulnerable to the same confound if instruction-following is scored on the same translated prompts or templates. The manuscript should (i) define how instruction-following is measured (separate probe vs. same PluraMath items), (ii) state whether instruction templates were language-adapted and how format consistency was enforced, and (iii) show that the association survives controls for residual translation quality or template mismatch. Otherwise the association cannot be cleanly attributed to model capability rather than prompt surface form.

    Authors: We agree the association is currently under-specified and could be confounded by prompt surface form. In the present draft, “instruction-following” was operationalized primarily as format compliance and successful answer extraction on the same PluraMath items (e.g., producing a parseable final answer in the required form), not via an independent probe. Instruction templates were language-adapted by native speakers as part of the same validation pipeline, with a shared answer-extraction schema, but we did not fully document adaptation rules or enforce/report format consistency metrics. In revision we will: (i) define instruction-following explicitly (format compliance / extractability rates, and any separate diagnostic if retained); (ii) document language adaptation of templates and the shared extraction/normalization rules; (iii) report the association after controlling for residual translation-quality indicators and template-mismatch flags (e.g., partial correlations or stratified analyses by validation residual band); and (iv) qualify the claim where it does not survive those controls. We will not overstate a causal capability story if the association is partly surface-form driven. The evaluation framework release will expose the exact templates and extraction code used. revision: yes

  3. Referee: Cross-language and cross-scale gap claims need explicit statistical support and matched evaluation conditions. The fine-grained analysis should report per-language and per-family scores with uncertainty (e.g., bootstrap CIs or paired tests against the PolyMath high-resource baseline under identical decoding and answer-extraction settings), and should document any language-specific answer-normalization rules. Without this, “persistent gap” remains a qualitative summary rather than a secured empirical result, especially for extreme low-resource languages where small absolute score differences can be dominated by extraction failures.

    Authors: We agree that “persistent gap” must be backed by uncertainty estimates and matched conditions, not only qualitative summary. All models were already run under a shared decoding and answer-extraction setup, but the manuscript did not report per-language/family uncertainty or formal comparisons to the PolyMath high-resource baseline, and language-specific normalization rules were under-documented. In revision we will: (1) report per-language and per-family scores with bootstrap confidence intervals; (2) add paired or stratified tests against the high-resource PolyMath baseline under identical decoding and extraction settings; (3) document all answer-normalization and extraction rules, including any language-specific exceptions (numeral systems, script variants, answer delimiters); and (4) separate extraction-failure rates from content-correct rates so that extreme low-resource gaps are not driven solely by unparseable outputs. We will revise figures/tables and the analysis section accordingly, and release the evaluation scripts that reproduce the CIs and tests. This converts the gap claim into a secured empirical result with explicit caveats where sample size or extraction noise dominates. revision: yes

Circularity Check

0 steps flagged

No significant circularity: PluraMath is an external benchmark construction and multi-model evaluation study with independent held-out scores.

full rationale

The paper constructs PluraMath by human-validated extension of the external PolyMath item pool to 18 underrepresented languages, then reports empirical accuracy of 27 third-party reasoning LLMs at four scales. Measured gaps and the association with instruction-following are outputs of independent model inference on held-out translated items; no parameter is fitted to a subset of PluraMath and then re-reported as a prediction, and no claimed first-principles quantity reduces by definition to an input equation. Citation of PolyMath (Wang et al., 2025) is ordinary dataset inheritance from non-overlapping authors and is not load-bearing for a uniqueness or derivation claim. The work is self-contained against external model benchmarks; residual concerns about translation equivalence affect correctness attribution, not circularity of the derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

As a dataset-and-benchmark paper, PluraMath rests mainly on domain assumptions about translation fidelity and problem equivalence rather than free parameters or invented physical entities. No fitted constants drive the central gap claim. The ledger therefore records the background assumptions required for score differences to be interpreted as multilingual reasoning gaps.

axioms (3)
  • domain assumption Native-speaker validation of machine translations yields items whose mathematical content and difficulty are equivalent to the source PolyMath problems across all 18 target languages.
    Required for attributing performance differences to model capability rather than translation distortion; invoked by the human-curated pipeline description.
  • domain assumption The selected 18 languages and six families are representative enough of mid-to-extreme low-resource settings for the reported gap to generalize beyond the sample.
    Language sampling is a design choice that shapes how broadly the 'persistent gap' claim can be read.
  • domain assumption Standard automatic or exact-match grading of model answers remains valid after translation (answer formats and numerical/symbolic equivalence hold).
    Benchmark scores depend on reliable answer checking; any language-specific formatting drift would confound results.

pith-pipeline@v0.9.1-grok · 6432 in / 2442 out tokens · 42967 ms · 2026-07-08T19:05:19.599476+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 51 canonical work pages · 22 internal anchors

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    work page 1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    work page 1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    work page doi:10.1145/322234.322243 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

    work page

  5. [5]

    Dan Gusfield , title =. 1997

    work page 1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    work page 2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

    work page

  8. [8]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe and Vineet Kosaraju and Mohammad Bavarian and Mark Chen and Heewoo Jun and Lukasz Kaiser and Matthias Plappert and Jerry Tworek and Jacob Hilton and Reiichiro Nakano and Christopher Hesse and John Schulman , title =. CoRR , volume =. 2021 , url =. 2110.14168 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  9. [9]

    The Twelfth International Conference on Learning Representations,

    Keiran Paster and Marco Dos Santos and Zhangir Azerbayev and Jimmy Ba , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    work page 2024

  10. [10]

    OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization

    Yiyou Sun and Shawn Hu and Georgia Zhou and Ken Zheng and Hannaneh Hajishirzi and Nouha Dziri and Dawn Song , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.18880 , eprinttype =. 2506.18880 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.18880 2025

  11. [11]

    MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs

    Jiakang Yuan and Tianshuo Peng and Yilei Jiang and Yiting Lu and Renrui Zhang and Kaituo Feng and Chaoyou Fu and Tao Chen and Lei Bai and Bo Zhang and Xiangyu Yue , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.21327 , eprinttype =. 2505.21327 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.21327 2025

  12. [12]

    CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?

    Tianwen Wei and Jian Luan and Wei Liu and Shuang Dong and Bin Wang , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2306.16636 , eprinttype =. 2306.16636 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2306.16636 2023

  13. [13]

    AGIEval:

    Wanjun Zhong and Ruixiang Cui and Yiduo Guo and Yaobo Liang and Shuai Lu and Yanlin Wang and Amin Saied and Weizhu Chen and Nan Duan , editor =. AGIEval:. Findings of the Association for Computational Linguistics:. 2024 , url =. doi:10.18653/V1/2024.FINDINGS-NAACL.149 , timestamp =

    work page doi:10.18653/v1/2024.findings-naacl.149 2024

  14. [14]

    Evaluating Large Language Models Trained on Code

    Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Pond. Evaluating Large Language Models Trained on Code , journal =. 2021 , url =. 2107.03374 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  15. [15]

    MathArena: Evaluating LLMs on Uncontaminated Math Competitions

    Mislav Balunovic and Jasper Dekoninck and Ivo Petrov and Nikola Jovanovic and Martin T. Vechev , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.23281 , eprinttype =. 2505.23281 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.23281 2025

  16. [16]

    CoRR , volume =

    Yiming Wang and Pei Zhang and Jialong Tang and Haoran Wei and Baosong Yang and Rui Wang and Chenshu Sun and Feitong Sun and Jiran Zhang and Junxuan Wu and Qiqian Cang and Yichang Zhang and Fei Huang and Junyang Lin and Fei Huang and Jingren Zhou , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2504.18428 , eprinttype =. 2504.18428 , timestamp =

    work page doi:10.48550/arxiv.2504.18428 2025

  17. [17]

    Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations , booktitle =

    Nuo Chen and Zinan Zheng and Ning Wu and Ming Gong and Dongmei Zhang and Jia Li , editor =. Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations , booktitle =. 2024 , url =. doi:10.18653/V1/2024.FINDINGS-EMNLP.411 , timestamp =

    work page doi:10.18653/v1/2024.findings-emnlp.411 2024

  18. [18]

    and Torralba, Antonio , booktitle =

    Alshammari, Shaden and Wen, Kevin and Zainal, Abrar and Hamilton, Mark and Safaei, Navid and Albarakati, Sultan and Freeman, William T. and Torralba, Antonio , booktitle =. 2026 , note =

    work page 2026

  19. [19]

    BenchMAX:

    Xu Huang and Wenhao Zhu and Hanxu Hu and Conghui He and Lei Li and Shujian Huang and Fei Yuan , editor =. BenchMAX:. Findings of the Association for Computational Linguistics:. 2025 , url =

    work page 2025

  20. [20]

    MMLU-ProX:

    Weihao Xuan and Rui Yang and Heli Qi and Qingcheng Zeng and Yunze Xiao and Aosong Feng and Dairui Liu and Yun Xing and Junjue Wang and Fan Gao and Jinghui Lu and Yuang Jiang and Huitao Li and Xin Li and Kunyu Yu and Ruihai Dong and Shangding Gu and Yuekang Li and Xiaofei Xie and Felix Juefei. MMLU-ProX:. Proceedings of the 2025 Conference on Empirical Met...

    work page doi:10.18653/v1/2025.emnlp-main.79 2025

  21. [22]

    A Survey of Multilingual Reasoning in Language Models , booktitle =

    Akash Ghosh and Debayan Datta and Sriparna Saha and Chirag Agarwal , editor =. A Survey of Multilingual Reasoning in Language Models , booktitle =. 2025 , url =

    work page 2025

  22. [23]

    Analyzing the Effects of Reasoning Types on Cross-Lingual Transfer Performance

    K, Karthikeyan and Sathe, Aalok and Aditya, Somak and Choudhury, Monojit. Analyzing the Effects of Reasoning Types on Cross-Lingual Transfer Performance. Proceedings of the 1st Workshop on Multilingual Representation Learning. 2021. doi:10.18653/v1/2021.mrl-1.8

    work page doi:10.18653/v1/2021.mrl-1.8 2021

  23. [24]

    CoRR , volume =

    Anaelia Ovalle and Candace Ross and Sebastian Ruder and Adina Williams and Karen Ullrich and Mark Ibrahim and Levent Sagun , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2512.22712 , eprinttype =. 2512.22712 , timestamp =

    work page doi:10.48550/arxiv.2512.22712 2025

  24. [25]

    Large Reasoning Models Are (Not Yet) Multilingual Latent Reasoners

    Yihong Liu and Raoyuan Zhao and Hinrich Sch. Large Reasoning Models Are (Not Yet) Multilingual Latent Reasoners , journal =. 2026 , url =. doi:10.48550/ARXIV.2601.02996 , eprinttype =. 2601.02996 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2601.02996 2026

  25. [26]

    Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks , booktitle =

    Fangru Lin and Shaoguang Mao and Emanuele La Malfa and Valentin Hofmann and Adrian de Wynter and Xun Wang and Si. Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks , booktitle =. 2025 , url =

    work page 2025

  26. [27]

    Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning , booktitle =

    Sara Rajaee and Rochelle Choenni and Ekaterina Shutova and Christof Monz , editor =. Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning , booktitle =. 2026 , url =

    work page 2026

  27. [28]

    Multilingual Reasoning via Self-training , booktitle =

    Leonardo Ranaldi and Giulia Pucci , editor =. Multilingual Reasoning via Self-training , booktitle =. 2025 , url =. doi:10.18653/V1/2025.NAACL-LONG.577 , timestamp =

    work page doi:10.18653/v1/2025.naacl-long.577 2025

  28. [29]

    CoRR , volume =

    Junxiao Liu and Zhijun Wang and Yixiao Li and Zhejian Lai and Liqian Huang and Xin Huang and Xue Han and Junlan Feng and Shujian Huang , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2602.05940 , eprinttype =. 2602.05940 , timestamp =

    work page doi:10.48550/arxiv.2602.05940 2026

  29. [30]

    Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Models , booktitle =

    Zhipeng Chen and Kun Zhou and Liang Song and Wayne Xin Zhao and Bingning Wang and Weipeng Chen and Ji. Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Models , booktitle =. 2025 , url =. doi:10.18653/V1/2025.EMNLP-MAIN.887 , timestamp =

    work page doi:10.18653/v1/2025.emnlp-main.887 2025

  30. [31]

    Translation as a Scalable Proxy for Multilingual Evaluation , journal =

    Sheriff Issaka and Erick Rosas Gonzalez and Lieqi Liu and Evans Kofi Agyei and Lucas Bandarkar and Nanyun Peng and David Ifeoluwa Adelani and Francisco Guzm. Translation as a Scalable Proxy for Multilingual Evaluation , journal =. 2026 , url =. doi:10.48550/ARXIV.2601.11778 , eprinttype =. 2601.11778 , timestamp =

    work page doi:10.48550/arxiv.2601.11778 2026

  31. [32]

    CoRR , volume =

    Sara Rajaee and Sebastian Vincent and Alexandre Berard and Marzieh Fadaee and Kelly Marchisio and Tom Kocmi , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2602.14763 , eprinttype =. 2602.14763 , timestamp =

    work page doi:10.48550/arxiv.2602.14763 2026

  32. [33]

    Qwen3 Technical Report

    Qwen Team , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.09388 , eprinttype =. 2505.09388 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025

  33. [34]

    CoRR , volume =

    Alexander Amini and Anna Banaszak and Harold Benoit and Arthur B. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2511.23404 , eprinttype =. 2511.23404 , timestamp =

    work page doi:10.48550/arxiv.2511.23404 2025

  34. [35]

    Scaling Latent Reasoning via Looped Language Models

    Rui. Scaling Latent Reasoning via Looped Language Models , journal =. 2025 , url =. doi:10.48550/ARXIV.2510.25741 , eprinttype =. 2510.25741 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2510.25741 2025

  35. [36]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    DeepSeek. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , journal =. 2025 , url =. doi:10.48550/ARXIV.2501.12948 , eprinttype =. 2501.12948 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.12948 2025

  36. [37]

    DeepSeek-V3 Technical Report

    DeepSeek. DeepSeek-V3 Technical Report , journal =. 2024 , url =. doi:10.48550/ARXIV.2412.19437 , eprinttype =. 2412.19437 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2412.19437 2024

  37. [38]

    Albert Q. Jiang and Alexandre Sablayrolles and Arthur Mensch and Chris Bamford and Devendra Singh Chaplot and Diego de Las Casas and Florian Bressand and Gianna Lengyel and Guillaume Lample and Lucile Saulnier and L. Mistral 7B , journal =. 2023 , url =. doi:10.48550/ARXIV.2310.06825 , eprinttype =. 2310.06825 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2310.06825 2023

  38. [39]

    Ministral 3

    Mistral AI , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2601.08584 , eprinttype =. 2601.08584 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2601.08584 2026

  39. [40]

    Gemma 3 Technical Report

    Gemma Team , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.19786 , eprinttype =. 2503.19786 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.19786 2025

  40. [41]

    Team OLMo and Pete Walsh and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Shane Arora and Akshita Bhagia and Yuling Gu and Shengyi Huang and Matt Jordan and Nathan Lambert and Dustin Schwenk and Oyvind Tafjord and Taira Anderson and David Atkinson and Faeze Brahman and Christopher Clark and Pradeep Dasigi and Nouha Dziri and Michal Guerquin and Hamis...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.00656 2025

  41. [42]

    gpt-oss-120b & gpt-oss-20b Model Card

    OpenAI , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2508.10925 , eprinttype =. 2508.10925 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.10925 2025

  42. [43]

    Kimi K2: Open Agentic Intelligence

    Kimi Team , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2507.20534 , eprinttype =. 2507.20534 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.20534 2025

  43. [44]

    Command A: An Enterprise-Ready Large Language Model

    Cohere , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2504.00698 , eprinttype =. 2504.00698 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2504.00698 2025

  44. [45]

    2025 , url =

    System. 2025 , url =

    work page 2025

  45. [46]

    OpenAI GPT-5 System Card

    OpenAI , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2601.03267 , eprinttype =. 2601.03267 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2601.03267 2026

  46. [47]

    Gemini: A Family of Highly Capable Multimodal Models

    Gemini Team , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2312.11805 , eprinttype =. 2312.11805 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2312.11805 2023

  47. [48]

    The Amazon Nova Family of Models: Technical Report and Model Card

    Amazon AGI , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.12103 , eprinttype =. 2506.12103 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.12103 2025

  48. [49]

    NLLB Team and Costa-juss \`a , Marta R. and Cross, James and C elebi, Onur and Elbayad, Maha and Heafield, Kenneth and Heffernan, Kevin and Kalbassi, Elahe and Lam, Janice and Licht, Daniel and Maillard, Jean and Sun, Anna and Wang, Skyler and Wenzek, Guillaume and Youngblood, Al and Akula, Bapi and Barrault, Loic and Gonzalez, Gabriel Mejia and Hansanti,...

    work page doi:10.1038/s41586-024-07335-x 2024

  49. [50]

    Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence

    NVIDIA , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2604.24954 , eprinttype =. 2604.24954 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.24954 2026

  50. [51]

    The State and Fate of Linguistic Diversity and Inclusion in the NLP World

    Joshi, Pratik and Santy, Sebastin and Budhiraja, Amar and Bali, Kalika and Choudhury, Monojit. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. doi:10.18653/v1/2020.acl-main.560

    work page doi:10.18653/v1/2020.acl-main.560 2020

  51. [52]

    Findings of the WMT 2025 Shared Task LLM s with Limited Resources for S lavic Languages: MT and QA

    Okabe, Shu and Dementieva, Daryna and Di Marco, Marion and Edman, Lukas and Haemmerl, Katharina and M e s kank, Marko and Hendrichowa, Anita and Fraser, Alexander. Findings of the WMT 2025 Shared Task LLM s with Limited Resources for S lavic Languages: MT and QA. Proceedings of the Tenth Conference on Machine Translation. 2025. doi:10.18653/v1/2025.wmt-1.27

    work page doi:10.18653/v1/2025.wmt-1.27 2025