Pith. sign in

REVIEW 2 major objections 6 minor 3 cited by

Crosslingual Reasoning through Test-Time Scaling

T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Scaling thinking tokens transfers English math reasoning to ten languages.

desk verdict The internal scaling result is real and useful, but the 'outperform models twice its size' headline is not yet supported because the big baselines never got the same test-time budget. read the letter →

arxiv 2505.05408 v1 pith:ZWYXJMNC submitted 2025-05-08 cs.CL cs.AIcs.LG

Zheng-Xin Yong , M. Farid Adilazuarda , Jonibek Mansurov , Ruochen Zhang , Niklas Muennighoff , Carsten Eickhoff , Genta Indra Winata , Julia Kreutzer
show 2 more authors
Stephen H. Bach Alham Fikri Aji
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords test-timescalingmultilingualreasoningchain-of-thoughtcrosslingualgeneralizationlow-resourcelanguageslanguageforcingMGSMmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether an English-centric reasoning model, one finetuned on a thousand English math problems with long chain-of-thought traces, can reason in other languages if it is simply given more thinking tokens at inference time. It finds that test-time scaling works across all ten MGSM languages for models of 3B parameters and above, with the 14B model outperforming models twice its size, such as DeepSeek-R1-Distill-Qwen-32B, on multilingual math. The gains are not just translation effects: the model's chain-of-thought stays mostly English but repeatedly quotes the non-English prompt and interprets it, a pattern the paper calls quote-and-think. Forcing the model to think in a low-resource language hurts accuracy and requires far more tokens, while thinking in high-resource languages is nearly as good as English and more efficient. The paper concludes that crosslingual test-time scaling is a viable lightweight baseline, but out-of-domain and low-resource reasoning remain unsolved.

What carries the argument

Budget forcing on long chain-of-thoughts: at each test, the model generates until it hits a maximum thinking-token cap, called truncation, or receives an appended 'Wait' token to continue reasoning, called extrapolation. This is the lever that converts extra inference FLOPs into accuracy. The complementary mechanism is quote-and-think, the dominant language-mixing pattern in which the model's mostly English reasoning quotes non-English phrases from the prompt and interprets them, which is what lets an English-trained reasoner parse non-English math problems.

What would settle it

Take a 1.5B and a 3B model from the same base family, finetune both on identical 1k English long-CoT samples, and measure MGSM accuracy at 500 versus 8,000 maximum thinking tokens; if the 1.5B model still fails to gain while the 3B model gains, the capacity threshold is confirmed, and if the 1.5B model gains too, the threshold is an artifact of the specific model comparison.

Watch

Extended reading notes

Core claim

The central claim is that English-only reasoning finetuning transfers across languages through test-time compute: increasing the maximum number of thinking tokens from 500 to 8,000 improves average MGSM accuracy for s1 models of 3B, 7B, 14B, and 32B parameters, including low-resource languages like Swahili and Telugu, and a 14B s1 model reaches 81.0% average accuracy, out-scoring DeepSeek-R1-Distill-Qwen-32B at 72.7% and Gemma-3-27B-it at 79.8%. The 1.5B model only gains 1.8 points, which the authors attribute to a capacity threshold rather than to a failure of test-time scaling. The paper also shows that the model's multilingual reasoning flows through a quote-and-think mechanism: English chains-of-thought lift non-English phrases from the question into quotation marks and reason about their meaning, a behavior present in 68.3% of the s1 training samples. When the reasoning language is forced, high-resource languages such as English, French, and German perform near English, while low-resource reasoning languages like Swahili and Telugu drop accuracy and need roughly 3.5 times more compute.

Load-bearing premise

The load-bearing premise is that the s1 model family represents English-centric reasoning models generally, so the observed 3B-parameter threshold is attributed to parameter count rather than to differences in base model, finetuning data scale, or training recipe between s1 and the 1.5B model it is compared against.

Editorial extensions

If this is right

  • Practitioners can use test-time scaling of English-centric reasoning models as a strong multilingual math baseline without any multilingual finetuning.
  • Data-efficient English finetuning with about 1,000 samples preserves multilingual ability better than large-scale distillation, reducing catastrophic forgetting for low-resource languages.
  • English-centric reasoning models should be allowed to reason in high-resource languages rather than forced into the query language, since in-language forcing in low-resource languages costs both accuracy and tokens.
  • Test-time scaling does not generalize to non-STEM domains; on cultural commonsense benchmarks, more thinking tokens can even reduce accuracy through overthinking.
  • A 3B-parameter threshold separates models that benefit from crosslingual test-time scaling from those that do not, contradicting earlier claims that much larger models would be needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 3B threshold is a general property of English-centric reasoning models, test-time scaling could make small multilingual models competitive without retraining, but the comparison does not isolate parameter count because s1 and DeepSeek-R1-1.5B differ in base model, training data scale, and recipe.
  • The quote-and-think pattern suggests an alternative to translation-based multilingual pipelines: instead of translating inputs, one could explicitly train or prompt models to quote and reinterpret the original phrase, preserving syntactic and cultural cues that translation would discard.
  • The negative correlation between token count and accuracy across reasoning languages implies that tokenizer fairness, not just model knowledge, shapes which languages are affordable to reason in, so better tokenizers could shift the Pareto frontier for low-resource languages.
  • A testable extension is to combine translated 'Wait' forcing with English-heavy reasoning for low-resource queries, letting English do the bulk of the computation while retaining the user's language in the output.
  • The paper does not test whether the same crosslingual scaling holds for non-mathematical STEM tasks such as code or formal proofs, where the quote-and-think pattern may behave differently.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper investigates whether test-time scaling of English-centric reasoning language models (RLMs), specifically the s1 family built from Qwen2.5-Instruct, transfers to multilingual mathematical reasoning. Using MGSM across 11 languages, the authors report that increasing the maximum thinking-token budget improves accuracy for s1 models of 3B parameters and above, with the largest gains at 14B, and that s1-14B outperforms several larger baselines including R1-Distill-Qwen-32B and Gemma-3-27B-it. The paper also documents a 'quote-and-think' language-mixing pattern in CoTs, studies methods for forcing reasoning into non-English languages, finds that reasoning in high-resource languages is more accurate and token-efficient than in low-resource languages, and reports limited out-of-domain generalization on Global-MMLU, FORK, and COPAL-ID. The authors conclude that data-efficient English reasoning finetuning plus test-time token scaling is a strong multilingual baseline, while cautioning about low-resource languages and out-of-domain tasks.

Significance. If the central claims hold, the paper provides a practical and low-cost recipe for multilingual math reasoning: English-only finetuning on 1k samples followed by test-time token scaling, and it adds evidence against the assumption that reasoning abilities are inherently English-bound. The quote-and-think analysis is a plausible and mechanistically interesting account of how English-centric RLMs parse non-English inputs, and the language-forcing experiments are a systematic contribution. The paper also includes useful cautionary results on overthinking and out-of-domain degradation. Strengths include the explicit train-test overlap check, the open release of code and models, and the breadth of ablations across model sizes and inference budgets. The main weakness is that the headline comparison to 'models twice its size' does not control for inference compute, and the attribution of prior negative results to model size alone rests on comparisons that confound multiple factors.

major comments (2)
  1. [Table 1 and Section 4.2] The claim that s1-14B 'outperforms models twice its size' is not supported as stated because the comparison is not controlled for inference compute. In Table 1, s1-14B uses extrapolation with an 8k maximum thinking-token budget and has an average generation length of 2352.3 tokens, while R1-Distill-Qwen-32B and Gemma-3-27B-it are evaluated 'without any inference budget constraint' and have average lengths of 1353.8 and 461.7 tokens, respectively. The Pareto frontier in Figure 1b covers only s1 and Qwen models, so it does not show where the larger baselines would sit if given comparable test-time compute. The observed gap could be partly or wholly due to asymmetric compute budgets rather than any superiority of the lightweight recipe. The authors should apply the same budget-forcing procedure (e.g., the same 'Wait' extrapolation or an 8k token cap) to the larger baselines and report the resulting accuracies, or qualify the claim accordingly.
  2. [Section 4.1 and Section 8] The conclusion that crosslingual test-time scaling requires at least 3B parameters, and that the negative findings of [17] are 'due to their usage of 1.5B models,' is under-supported because the comparison confounds parameter count with model family, base model, training-data scale, and recipe. The s1-1.5B model and the R1-Distill-Qwen-1.5B model used in [17] differ in base model, in finetuning data size (1k vs. 800k samples), and in training procedure. The observed 3B threshold is therefore established only for the s1 family on Qwen2.5 bases. The authors should either test additional RLM families at 1.5B and 3B, or explicitly limit the threshold claim to s1 models.
minor comments (6)
  1. [Section 4.1] There is a typo in 'two main obsevations' (should be 'observations').
  2. [Section 3] The sentence ending 'lm-evaluation-harness library [47] as the main evaluation framework.3.' contains a stray '3.' that should be removed.
  3. [Box 2] The caption says 'CoTs prdocued' (should be 'produced').
  4. [Section 6.3] The sentence 'we force the model to all M possible languages' is grammatically incomplete; it should read 'we force the model to reason in all M possible languages.'
  5. [Figure 5] The reported correlation of -0.811 between token count and accuracy is based on only 11 language-level aggregate points; a confidence interval or a per-item analysis would make the strength of the claim clearer.
  6. [Section 4.2] The sentence listing prior work, 'all prior state-of-the-art models that involve finetuning on multilingual data such as MetaMath, MAPO, and [52, 37, 49, 35]', is awkwardly phrased and the citation [52] repeats the MetaMath reference; consider rewriting for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all central claims are direct empirical measurements on open models; no parameter is fitted to the target result.

full rationale

The paper's claims—crosslingual test-time scaling, the quote-and-think language-mixing pattern, language-forcing effects, and domain generalization—are each evaluated rather than derived. The s1 models are taken from prior open-source work [9], and the paper measures their MGSM accuracy, token counts, and language-mixing labels; there is no equation whose output is an input. The claim that [17]'s negative findings are due to the 1.5B-parameter scale is an explanatory comparison, not a reduction: the paper's own 1.5B/3B/7B/14B/32B s1 results are independent evidence of a size trend, although the comparison to R1-1.5B does not isolate parameter count from recipe differences (a correctness limitation, not circularity). The 'outperforms models twice its size' headline is based on Table 1, where larger baselines were not given the same inference budgets; this is an incomplete control, but it is not a circular step because no quantity is defined in terms of the claimed result or fitted to it. The quote-and-think analysis (Section 5.2 and Appendix C.4) explains the behavior by pointing to 68.3% of s1's English training samples exhibiting the same pattern; this is an external training-data prior, not a self-referential derivation. The paper also explicitly credits the linguistic term 'foreign-language quotation' [54], so the renamed pattern is not presented as a new derivation. No fitted parameters, no uniqueness theorems, and no load-bearing self-citation chains appear. Consequently the paper is self-contained as an empirical study; any weaknesses are matters of experimental control and external validity, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities; the paper is an empirical evaluation. The axioms are standard evaluation assumptions plus one non-trivial assumption about how representative s1 is for the 3B threshold.

assumptions (4)
  • domain assumption MGSM accuracy is a valid proxy for multilingual mathematical reasoning ability.
    The paper's main conclusions about crosslingual test-time scaling and language forcing rely on aggregate accuracy over 250 translated math problems per language in MGSM, which is a standard but narrow benchmark.
  • ad hoc to paper The s1 model family is representative of English-centric RLMs.
    The 3B threshold for scaling benefit is derived from the s1 family alone; the comparison to [17] uses a different base model and 800k-sample finetuning, so the parameter threshold is not isolated.
  • domain assumption GPT-4o-mini correctly extracts multiple-choice answers from generated responses for Global-MMLU.
    Cross-domain results depend on an external LLM parser (Appendix E.1); errors in extraction would affect the reported accuracy.
  • domain assumption Automatic language identification (lingua, stanza) correctly detects language-mixing patterns in CoTs.
    The quote-and-think analysis in Section 5 relies on these tools; the authors exclude Latin-script languages to avoid misclassification, which limits but does not invalidate the claim for the four studied languages.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Crosslingual Reasoning through Test-Time Scaling." pith.science (2026). https://pith.science/paper/ZWYXJMNC

@misc{pith2026250505408,
  author       = {Pith},
  title        = {Pith review of: Crosslingual Reasoning through Test-Time Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZWYXJMNC}},
  note         = {Machine review of arXiv:2505.05408}
}
read the original abstract

Reasoning capabilities of large language models are primarily studied for English, even when pretrained models are multilingual. In this work, we investigate to what extent English reasoning finetuning with long chain-of-thoughts (CoTs) can generalize across languages. First, we find that scaling up inference compute for English-centric reasoning language models (RLMs) improves multilingual mathematical reasoning across many languages including low-resource languages, to an extent where they outperform models twice their size. Second, we reveal that while English-centric RLM's CoTs are naturally predominantly English, they consistently follow a quote-and-think pattern to reason about quoted non-English inputs. Third, we discover an effective strategy to control the language of long CoT reasoning, and we observe that models reason better and more efficiently in high-resource languages. Finally, we observe poor out-of-domain reasoning generalization, in particular from STEM to cultural commonsense knowledge, even for English. Overall, we demonstrate the potentials, study the mechanisms and outline the limitations of crosslingual generalization of English reasoning test-time scaling. We conclude that practitioners should let English-centric RLMs reason in high-resource languages, while further work is needed to improve reasoning in low-resource languages and out-of-domain contexts.

Figures

Figures reproduced from arXiv: 2505.05408 by the authors.

Figure 1
Figure 1. Crosslingual test-time scaling of s1 and Qwen models on the MGSM benchmark ( [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Proportion of dominant languages in models’ entire responses when queried with multilingual math questions. “same” in￾dicates that the response language is the same as query language. 92.5 95.0 97.5 100.0 ja ru th zh 0.0 2.5 5.0 Percentage (%) English-only quote-and-think intersentential intrasentential [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Language and domain breakdown for Global-MMLU benchmark. Dashed lines indicate [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: MGSM accuracy against number of thinking tokens in s1 models’ outputs in different reasoning languages. Inference cost analysis Our analysis of infer￾ence costs across reasoning languages in Fig￾ure 5 reveals a significant negative correlation (-0.811) between token co…
Figure 6
Figure 6. Figure 6: Effects of thinking time for s1 models on different domains of Global-MMLU bench [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Performance comparison between s1 and zero-shot prompting of Qwen baseline for [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Proportion of dominant languages used by 14B-sized models’ responses when queried with [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Breakdown of s1-32B’s finegrained intrasentential language mixing patterns. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It

    cs.CL 2025-05 accept novelty 6.0 of 10

    LLM safety research at ACL venues from 2020 to 2024 is predominantly English-only, and the language gap is growing over time.

  2. ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark

    cs.CV 2025-05 conditional novelty 6.0 of 10

    ARB provides 1,356 Arabic multimodal questions with 5,119 human-reviewed reasoning steps and shows leading models score much higher on reasoning fluency than on correct answers.

  3. Cost of Reasoning in non-English Languages: A Case Study on Japanese

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Japanese reasoning-language control is feasible with CPT plus GRPO, but incurs a capability cost and does not free-improve cultural Japanese performance.

Reference graph

Works this paper leans on

78 extracted references · 35 canonical work pages · cited by 3 Pith papers

  1. [17]

    Linguistic generalizability of test-time scaling in mathematical reasoning

    Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne. Linguistic generalizability of test-time scaling in mathematical reasoning. arXiv preprint arXiv:2502.17407, 2025

    arXiv 2025

  2. [1]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    arXiv 2024

  3. [2]

    Large language monkeys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    arXiv 2024

  4. [3]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024

    arXiv 2024

  5. [4]

    A simple model of inference scaling laws

    Noam Levi. A simple model of inference scaling laws. arXiv preprint arXiv:2410.16377, 2024

    arXiv 2024

  6. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  7. [6]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. 13

    arXiv 2024

  8. [7]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. Technical report, OpenAI, April 2025

    work page 2025

Show all 78 references
  1. [8]

    Towards reasoning era: A survey of long chain-of-thought for reasoning large language models

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wangxiang Che. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025

    2025 arXiv

  2. [9]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  3. [10]

    Limo: Less is more for reasoning

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025

    2025 arXiv

  4. [11]

    The multilingual mind : A survey of multilingual reasoning in language models, 2025

    Akash Ghosh, Debayan Datta, Sriparna Saha, and Chirag Agarwal. The multilingual mind : A survey of multilingual reasoning in language models, 2025

    2025

  5. [12]

    Advancing language model reasoning through reinforcement learning and inference scaling

    Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. Advancing language model reasoning through reinforcement learning and inference scaling. arXiv preprint arXiv:2501.11651, 2025

    2025 arXiv

  6. [13]

    Training large language models to reason in a continuous latent space

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  7. [14]

    Critic: Large language models can self-correct with tool-interactive critiquing

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. In ICLR, 2024

    2024

  8. [15]

    Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though

    Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though. arXiv preprint arXiv:2501.04682, 2025

    2025 arXiv

  9. [16]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  10. [18]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in N...

    2022

  11. [19]

    Program induction by rationale generation: Learning to solve and explain algebraic word problems

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Li...

  12. [20]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  13. [21]

    Show your work: Scratchpads for intermediate computation with language models

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021

    2021 arXiv

  14. [22]

    Chain of thought empowers transformers to solve inherently serial problems

    Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. In The Twelfth International Conference on Learning Representations, 2024. 14

    2024

  15. [23]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307, 2025

    2025 arXiv

  16. [24]

    Evolving deeper llm thinking

    Kuang-Huei Lee, Ian Fischer, Yueh-Hua Wu, Dave Marwood, Shumeet Baluja, Dale Schu- urmans, and Xinyun Chen. Evolving deeper llm thinking. arXiv preprint arXiv:2501.09891, 2025

    2025 arXiv

  17. [25]

    O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489, 2024

    Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489, 2024

    2024 arXiv

  18. [26]

    Bespoke-stratos: The unreasonable effectiveness of reasoning distilla- tion

    Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distilla- tion. www.bespokelabs.ai/blog/bespoke-stratos-the-unreasonable-effectiveness-of-reasoning- distillation, 2025. Accessed: 2025-01-22

    2025

  19. [27]

    Millions scale dataset distilled from r1-32b

    Sathwik Tejaswi Madhusudhan, Shruthan Radhakrishna, Jash Mehta, and Toby Liang. Millions scale dataset distilled from r1-32b. https://huggingface.co/datasets/ServiceNow-AI/R1-Distill- SFT, 2025

    2025

  20. [28]

    Tina: Tiny reasoning models via lora

    Shangshang Wang, Julian Asilis, Ömer Faruk Akgül, Enes Burak Bilgin, Ollie Liu, and Willie Neiswanger. Tina: Tiny reasoning models via lora. arXiv preprint, April 2025

    2025

  21. [29]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  22. [30]

    Think before you speak: Training language models with pause tokens

    Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations, 2024

    2024

  23. [31]

    Language models are multilingual chain-of-thought reasoners

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush V osoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on L...

    2023

  24. [32]

    Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages

    Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natura...

    2023

  25. [33]

    Question translation training for better multilingual reasoning

    Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. Question translation training for better multilingual reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computational Linguistics: ACL 2024, ...

    2024

  26. [34]

    Understand, solve and translate: Bridging the multilingual mathematical reasoning gap

    Hyunwoo Ko, Guijin Son, and Dasol Choi. Understand, solve and translate: Bridging the multilingual mathematical reasoning gap. arXiv preprint arXiv:2501.02448, 2025

    2025 arXiv

  27. [35]

    LangBridge: Multilingual reasoning without multilingual supervision

    Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat, and Minjoon Seo. LangBridge: Multilingual reasoning without multilingual supervision. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for...

    2024

  28. [36]

    Mindmerger: Efficiently boosting LLM reasoning in non-english languages

    Zixian Huang, Wenhao Zhu, Gong Cheng, Lei Li, and Fei Yuan. Mindmerger: Efficiently boosting LLM reasoning in non-english languages. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 15

    2024

  29. [37]

    MAPO: Advancing multilingual reasoning through multilingual-alignment-as-preference optimization

    Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. MAPO: Advancing multilingual reasoning through multilingual-alignment-as-preference optimization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual...

    2024

  30. [38]

    Language imbalance driven rewarding for multilingual self-improving

    Wen Yang, Junhong Wu, Chen Wang, Chengqing Zong, and Jiajun Zhang. Language imbalance driven rewarding for multilingual self-improving. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  31. [39]

    Breaking language barriers in multilingual mathematical reasoning: Insights and observations

    Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. arXiv preprint arXiv:2310.20246, 2023

    2023 arXiv

  32. [40]

    Language model developers should report train-test overlap

    Andy K Zhang, Kevin Klyman, Yifan Mai, Yoav Levine, Yian Zhang, Rishi Bommasani, and Percy Liang. Language model developers should report train-test overlap. arXiv preprint arXiv:2410.08385, 2024

    2024 arXiv

  33. [41]

    Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T....

    2024

  34. [42]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  35. [43]

    FORK: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models

    Shramay Palta and Rachel Rudinger. FORK: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 9952...

    2023

  36. [44]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of ...

    2019

  37. [45]

    COPAL- ID: Indonesian language reasoning with local culture and nuances

    Haryo Wibowo, Erland Fuadi, Made Nityasya, Radityo Eko Prasojo, and Alham Aji. COPAL- ID: Indonesian language reasoning with local culture and nuances. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American Chapter of t...

    2024

  38. [46]

    Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90–95, 2011

    2011

  39. [47]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  40. [48]

    Mammoth: Building math generalist models through hybrid instruction tuning

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023. 16

    2023 arXiv

  41. [49]

    SLAM: Towards efficient multilingual reasoning via selective language alignment

    Yuchun Fan, Yongyu Mu, YiLin Wang, Lei Huang, Junhao Ruan, Bei Li, Tong Xiao, Shujian Huang, Xiaocheng Feng, and Jingbo Zhu. SLAM: Towards efficient multilingual reasoning via selective language alignment. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barba...

    2025

  42. [50]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  43. [51]

    Beyond chinchilla-optimal: Accounting for inference in language model scaling laws

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.00448, 2023

    2023 arXiv

  44. [52]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  45. [53]

    Qwen3: Think deeper, act faster, 4 2025

    Qwen Team. Qwen3: Think deeper, act faster, 4 2025. 2036 words, 10 min read

    2025

  46. [54]

    Foreign-language quotations and code-switching: The grammar behind

    Philippe De Brabanter. Foreign-language quotations and code-switching: The grammar behind. In ESSE Conference (European Society for the Study of English), 2004

    2004

  47. [55]

    The state and fate of linguistic diversity and inclusion in the NLP world

    Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of th...

    2020

  48. [56]

    Language model tokenizers introduce unfairness between languages

    Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. Advances in neural information processing systems, 36:36963–36990, 2023

    2023

  49. [57]

    Smith, and Yulia Tsvetkov

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R Mortensen, Noah A. Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  50. [58]

    Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai

    Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, Yikai Zhang, Yuqing Yang, Ting Wu, Binjie Wang, Shichao Sun, Yang Xiao, Yiyuan Li, Fan Zhou, Steffi Chern, Yiwei Qin, Yan Ma, Jiadi Su, Yixiu Liu, Yuxian...

    2024

  51. [59]

    Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse

    Ryan Liu, Jiayi Geng, Addison J Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L Griffiths. Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse. arXiv preprint arXiv:2410.21333, 2024

    arXiv

  52. [60]

    The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks

    Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235, 2025

    2025 arXiv

  53. [61]

    Do not think that much for 2+ 3=? on the overthinking of o1-like llms

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187, 2024. 17

    2024 arXiv

  54. [62]

    Stop overthinking: A survey on efficient reasoning for large language models

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Hanjie Chen, Xia Hu, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419, 2025

    2025 arXiv

  55. [63]

    BLOOM+1: Adding language support to BLOOM for zero-shot prompting

    Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, and Vassilina Nikoulina. BLOOM+1: Adding la...

    2023

  56. [64]

    Understanding catastrophic forgetting in language models via implicit inference

    Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forgetting in language models via implicit inference. In The Twelfth International Conference on Learning Representations, 2024

    2024

  57. [65]

    Translating across cultures: LLMs for intralingual cultural adaptation

    Pushpdeep Singh, Mayur Patidar, and Lovekesh Vig. Translating across cultures: LLMs for intralingual cultural adaptation. In Libby Barak and Malihe Alikhani, editors, Proceedings of the 28th Conference on Computational Natural Language Learning, pages 400–418, Miami, FL, USA, ...

    2024

  58. [66]

    Multilingual!= multicultural: Evalu- ating gaps between multilingual capabilities and cultural alignment in llms

    Jonathan Rystrøm, Hannah Rose Kirk, and Scott Hale. Multilingual!= multicultural: Evalu- ating gaps between multilingual capabilities and cultural alignment in llms. arXiv preprint arXiv:2502.16534, 2025

    2025 arXiv

  59. [67]

    Mortensen, and Graham Neubig

    Nathaniel Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. ChatGPT MT: Competitive for high- (but not low-) resource languages. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Transla- tion, pa...

    2023

  60. [68]

    Shortcomings of LLMs for low-resource translation: Retrieval and understanding are both the problem

    Sara Court and Micha Elsner. Shortcomings of LLMs for low-resource translation: Retrieval and understanding are both the problem. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz, editors, Proceedings of the Ninth Conference on Machine Translation, pages 1332–1354,...

    2024

  61. [69]

    Is llm the silver bullet to low-resource languages machine translation? arXiv preprint arXiv:2503.24102, 2025

    Yewei Song, Lujun Li, Cedric Lothritz, Saad Ezzini, Lama Sleem, Niccolo Gentile, Radu State, Tegawendé F Bissyandé, and Jacques Klein. Is llm the silver bullet to low-resource languages machine translation? arXiv preprint arXiv:2503.24102, 2025

    2025 arXiv

  62. [70]

    LLM-powered data augmen- tation for enhanced cross-lingual performance

    Chenxi Whitehouse, Monojit Choudhury, and Alham Fikri Aji. LLM-powered data augmen- tation for enhanced cross-lingual performance. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, p...

    2023

  63. [71]

    LexC-gen: Generating data for extremely low-resource languages with large language models and bilingual lexicons

    Zheng Xin Yong, Cristina Menghini, and Stephen Bach. LexC-gen: Generating data for extremely low-resource languages with large language models and bilingual lexicons. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational L...

    2024

  64. [72]

    XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models

    Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2...

    2023

  65. [73]

    Adapters for altering LLM vocabularies: What languages benefit the most? In The Thirteenth International Conference on Learning Representations, 2025

    HyoJung Han, Akiko Eriguchi, Haoran Xu, Hieu Hoang, Marine Carpuat, and Huda Khayrallah. Adapters for altering LLM vocabularies: What languages benefit the most? In The Thirteenth International Conference on Learning Representations, 2025. 18

    2025

  66. [74]

    ByT5: Towards a token-free future with pre-trained byte-to-byte models

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306, 2022

    2022

  67. [75]

    Small models struggle to learn from strong reasoners

    Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ra- masubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. arXiv preprint arXiv:2502.12143, 2025

    2025

  68. [76]

    How do multilingual models remember? investigating multilingual factual recall mechanisms

    Constanza Fierro, Negar Foroutan, Desmond Elliott, and Anders Søgaard. How do multilingual models remember? investigating multilingual factual recall mechanisms. arXiv preprint arXiv:2410.14387, 2024

    2024 arXiv

  69. [77]

    Sie isst 3 Eier zum Frühstück und verwendet 4 Eier für Muffins, also verwendet sie insgesamt 3 + 4 = 7 Eier pro Tag

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Pr...

    2024

  70. [2017]

    Association for Computational Linguistics

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.