Pith. sign in

REVIEW 4 major objections 8 minor 29 references

Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap

T0 review · 4 major / 8 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The native-versus-translate gap on MGSM is a token-budget artifact, not a stable measure of reasoning ability.

desk verdict A careful, mostly convincing demonstration of budget-regime dependence in the MGSM gap, with the exact magnitudes conditional on an unvalidated length normalizer. read the letter →

arxiv 2608.04160 v1 pith:TAYZGEGC submitted 2026-08-04 cs.CL cs.LG

Ankit Goyal , Jaideep Ray This is my paper · ORCID
classification cs.CLcs.LG
keywords multilingualreasoninggaptokenbudgetoutputcapMGSMFLORES-200chain-of-thoughtlengthnormalizationsweep
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the native-versus-translate gap on the MGSM reasoning benchmark is substantially a token-budget artifact: languages need different numbers of tokens to express the same content, so a fixed output cap penalizes the native language. The gap is shown to equal the gain in native accuracy obtained by scaling the cap by the language's token premium; at a large frozen budget the gap is near zero (0.00–0.15 points for Qwen), while at tight caps it peaks at 15.0–38.9 points, and those peaks were prospectively confirmed on 540,000 independently hard-capped decodes. If the paper is right, single-budget accuracy measurements do not give a stable estimate of the multilingual reasoning gap, and evaluators should report accuracy across the budget regime rather than at one cap.

What carries the argument

The load-bearing object is the estimand identity $\Delta_L(B) = \mathrm{acc}_N(\lfloor r_{m,L} B \rfloor) - \mathrm{acc}_N(B)$, with $r_{m,L}$ the FLORES-200 token premium: the ratio of tokens the model's own tokenizer needs for parallel devtest content in language $L$ versus English. This reduces the gap to a finite increment of the native accuracy curve, so the gap's peak location follows the native answer-emission distribution and its height depends on the premium-scaled interval. A companion identity, $\Delta_L(B) = G(\lfloor r B \rfloor) - G(B)$ with $G(t) = P(C = 1, E \le t)$ the correct-emission sub-CDF, predicts the peak from one long-cap run to within 0.65 points on the three pre-specified MGSM cells. The empirical machinery is the ledger of stored 4096-token generations scored at every prefix budget, paired with an independent sample in which every budget is decoded under its own hard cap.

What would settle it

Retake the Qwen Swahili peak (discovered at $B = 128$) but scale the native budget by the measured behavioral trace-length ratio (1.179) instead of the FLORES premium (1.94); the paper's analysis predicts the artifact falls below 5 points. If independently hard-capped decodes at the trace-length-scaled cap still show a gap above 5 points, the premium-based budget-artifact explanation for that cell would be wrong.

Watch

Extended reading notes

Core claim

The central claim is that the measured native-vs-translate gap on MGSM is not a stable quantity but a function of the output-token budget, formalized by the identity $\Delta_L(B) = \mathrm{acc}_N(\lfloor r_{m,L} B \rfloor) - \mathrm{acc}_N(B)$, where $r_{m,L}$ is the FLORES-200 token premium of language $L$ over English for model $m$. The gap is therefore just how many more native answers become correct when the native budget is scaled by the premium; the translate arm cancels out. At the frozen budget $B^* = 1024$ the gap is near zero for all three Qwen languages and all six confirmatory tests fail to reject, whereas at tight caps the gap reaches 34.2 (German), 38.9 (Thai), and 15.0 points (Swahili), and these three peaks survive on an independent sample of 540,000 hard-capped decodes. A third family of tests varies only the announced budget at a fixed enforced cap and finds that announcing 128 rather than 2048 tokens moves Thai native accuracy by 5.1 points, showing that accuracy is not a function of the enforced cap alone. The residual difference above saturation is interpreted as a strategy-performance gap, not an identified reasoning deficit.

Load-bearing premise

The load-bearing premise is that the FLORES-200 token premium, computed on parallel devtest sentences, is the right length normalizer for MGSM reasoning traces; if the true trace-length ratio differs, the paper's own sensitivity analysis shows the artifact's magnitude and even its existence change in specific cells (Qwen Swahili depends on the premium exceeding the behavioral ratio of 1.254 versus 1.179).

Editorial extensions

If this is right

  • Single-budget accuracy estimates of the multilingual reasoning gap are unstable: the measured gap swings by up to 57 points across budgets and can even reverse which strategy scores higher at tight caps.
  • Length-normalizing the native budget with the FLORES-200 premium moves the gap by up to 38.9 points where the cap binds, and the effect vanishes once native accuracy saturates.
  • Vocabulary extension closes the gap only where the cap still truncates: 0.0 points at the frozen budget, up to 4.9 points where 19% of traces still truncate.
  • Accuracy is not a function of the enforced cap alone: announcing a budget changes behavior at a fixed cap, so a deployment that discloses its budget must evaluate under the disclosure it ships.
  • Evaluations should treat the output cap as an independent variable and report accuracy across a budget sweep, marking the region where answer emission binds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the budget-artifact account generalizes, prior single-budget estimates of multilingual reasoning gaps on other benchmarks should be re-read with suspicion; a minimal re-analysis would report native accuracy at $B$ and at the premium-scaled cap, which is enough to bound the artifact.
  • The announcement effect suggests that budget disclosure is a behavioral variable in deployed systems: the same enforced cap can yield different accuracy depending on what the prompt says, so benchmark comparisons that include the cap in the prompt are comparing more than the model's reasoning.
  • The correct-emission sub-CDF identity offers a cheap diagnostic for choosing evaluation budgets: one long-cap run can locate where the budget artifact peaks, so future work could pre-register peak budgets from a pilot run instead of sweeping many budgets.
  • The vocabulary-extension result implies that token-count interventions are only as valuable as the answer-emission tail they rescue; a production team deciding between a bigger cap and a language-specific tokenizer should first measure how much accuracy longer prefixes actually recover on its own traces.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes treating the output-token cap as an independent variable in multilingual evaluation. It defines a length-normalized estimand Δ_L(B)=accN(⌊r_{m,L}B⌋)−accN(B), where r is the FLORES-200 token premium, so the native-vs-translate gap becomes a finite increment of the native accuracy curve. On MGSM with Qwen3-8B and Llama-3.1-8B-Instruct, it reports that the gap is near zero at the frozen B*=1024 (Qwen de 0.00, th 0.15, sw 0.05) while peaking at tight budgets (de 34.2, th 38.9, sw 15.0), with peaks replicated on 540,000 independent hard-capped decodes. It also reports that announcing a budget changes behavior at a fixed enforced cap (one significant cell, Thai NATIVE +5.1), that vocabulary extension closes the gap only where truncation still binds, and that a correct-emission sub-CDF identity from one long-cap run tracks the peaks. The paper is explicitly methodology-focused, with audits on language ID, translation quality, parser robustness, decoder parity, and normalizer sensitivity.

Significance. If correct, the paper's central claim is important: single-budget accuracy measurements do not provide a stable estimate of the multilingual reasoning gap, and the cap should be swept and length-normalized. The paper has genuine methodological strengths: the estimand is an identity, the confirmatory families were pre-specified and frozen, the independent hard-capped replication is substantial (540k decodes), the six-test Holm family is reported with tail-conservatism calibration, and the paper is unusually transparent about its own circularity and limitations (e.g., the sub-CDF agreement is explicitly flagged as agreeing by construction with replay and is checked against independent decodes). The announcement experiment is a clean design that isolates disclosure from truncation. The adaptation triage is honestly scoped as a heuristic. These strengths make the paper publishable if the normalizer concern is resolved or adequately bounded.

major comments (4)
  1. [§4.2/Appendix D] The central quantitative claim is conditional on a load-bearing normalizer. Equation (1) defines the budget artifact entirely through the FLORES-200 premium r_{m,L}, but Appendix D shows the Qwen Swahili 5-point artifact disappears if the observed behavioral trace-length ratio (1.179) is used instead of the FLORES premium (threshold 1.254), and Llama Thai never reaches 5 points even at 1.5×FLORES. The Limitations explicitly state that the same-content trace-premium validation is outstanding. Please either supply that validation (e.g., measuring token premiums on parallel MGSM item content in both languages for the exact NATIVE/TRANSLATE-ACT traces) or relegate the Swahili 5-point claim and any magnitude claims that inherit the FLORES premium to exploratory status; the qualitative conclusion that budget regimes change the measured gap is not affected by this issue.
  2. [Limitations/§4.4] The announcement experiment's headline cell (Thai NATIVE +5.1 points) is one significant cell out of four, and the direction is described as 'announcing a tighter budget raised accuracy', which the paper reports honestly. However, the conclusion in the abstract that 'accuracy is not a function of the enforced cap alone' is stated as a general consequence of one significant cell. I would ask for a more careful wording: the data support an effect in one cell, not a general statement, though the monotone dose-response in that cell (63.20/59.90/58.10) is suggestive. This is a presentation issue but it matters for how the abstract is read.
  3. [§6/Appendix G] The adaptation triage conclusion that the vocabulary extension closes the gap only where truncation binds is supported by the data, but the scope caveats in Appendix G weaken the generality more than the main text acknowledges: the extension is trained on NATIVE traces only, the English control is aggregate (not per-string), the fold split is fixed, and G(4096) is the largest stored prefix, not a non-binding regime. Since the section is explicitly scoped as a heuristic, this is not a blocking issue, but I would ask that the first two caveats be moved into the main text of §6, because Table 3 is easily over-read as a general cost-benefit result.
  4. [§4.1/Appendix E] The tail-conservatism factor is described as 'a conservative safeguard rather than a verified family-wise calibration', and the corrected type-I rate (0.00917) exceeds the target (0.00833). This is an honest and unusual disclosure, but the reader should be told whether the 1.3 factor was pre-specified before seeing the data or chosen after observing the anti-conservatism. The text says 'pre-specified 1.3× tail-conservatism factor' but also says calibration 'found mild anti-conservatism', which is ambiguous. Please clarify the chronology; if the factor was chosen after the fact, the confirmatory status of the H1 and H3 tests is weaker than presented.
minor comments (8)
  1. [Abstract] The phrase 'length normalization moves it by up to 38.9 points where the cap binds' is a replay-ledger quantity; the abstract later mentions 540,000 independent decodes, so consider clarifying that the 38.9 is from the stored-prefix sweep, not the independent sample.
  2. [§4.1] The phrase 'This procedureally matched secondary analysis' has a typo ('procedureally' for 'procedurally') and the sentence 'also rejects nothing' is oddly phrased given that the test was not designed to reject anything; consider 'also yields no rejection'.
  3. [Table 1 / §4.2] The Swahili row of Table 1 cites NATIVE accuracy 8.70 versus TRANSLATE-ACT 0.60 at B=128, a degenerate floor cell. The main text cautions about this, but the table note should repeat it because the table alone is misleading without it.
  4. [Appendix D] The sentence 'Qwen Swahili is the exception: its threshold of 1.254 sits above the behavioral ratio of 1.179' is load-bearing and should be echoed in §4.2, not only in the appendix, for the reasons in my major comment 1.
  5. [§6] The sentence 'We cannot price the prompting rung, because adopting TRANSLATE-ACT closes G by definition' is confusing: adopting TRANSLATE-ACT replaces the NATIVE arm rather than being an incremental rung. Please rephrase to say that the comparator is the strategy itself, so it cannot be priced as a separate intervention.
  6. [Appendix H] 'capped_eos false' and 'capped_eos true' are used without definition; define them at first use.
  7. [References] Several 2026 preprints appear in the reference list without in-text citation numbers (e.g., Lasbordes et al., Liu et al., Su et al., Silvestri and Cetin, Zhang et al.). Please reconcile citations and list entries.
  8. [§5/Limitations] The relationship between the exploratory blind LLM adjudication and the outstanding frozen human GlotLID validation should be stated once in one place; currently the reader must piece it together from §5 and the Limitations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central identity is explicitly acknowledged as by-construction and is validated against independent decodes, while the FLORES premium is an external input with an acknowledged sensitivity caveat.

full rationale

The derivation chain is self-contained and the paper is unusually careful about the boundary between identity and evidence. Equation (1) is an algebraic identity after TRANSLATE-ACT cancels, and the paper does not treat replay-ledger agreement as evidence: Appendix J explicitly states that 'Agreement with REPLAY deltas is not evidence: under absorbing correctness, the sub-CDF on a replay ledger is algebraically identical to the replay accuracy difference and therefore agrees by construction.' The main confirmatory claims rest on 540,000 independently hard-capped decodes with pre-specified frozen budgets, not on the stored-prefix ledger alone. The FLORES-200 premium is obtained from an external public parallel benchmark using each model's own tokenizer; it is not fitted to the MGSM target result. Appendix D probes normalizer sensitivity and shows that one cell (Qwen Swahili) depends on the FLORES premium being above the behavioral trace-length ratio, and the Limitations section states that 'same-content trace-premium validation remain[s] outstanding.' This is an acknowledged assumption and a correctness risk, but not a circular reduction: the premium is not derived from the quantity it is used to explain. The vocabulary-extension rung is cross-fitted over disjoint item halves so that no evaluated item contributes to the tokenizer that scores it, and the paper explicitly declines to price the prompting rung because it 'closes G by construction.' No load-bearing self-citation or imported uniqueness theorem appears; the one citable benchmark lineage (FLORES/NLLB) is standard external infrastructure. The only by-construction identity in the paper is flagged as such and is checked against genuinely separate decoding outcomes, so the central claim is not reduced to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper is a measurement study, not a model-fitting derivation. Its free parameters are mostly pre-specified statistical thresholds and the external FLORES-200 premiums that define the length-matched estimand. The load-bearing domain assumptions are the transferability of the FLORES premium to MGSM reasoning traces, the validity of prefix-defined evaluation away from the six confirmed cells, approximate answer-absorption, and the reliability of automated language labels. No new physical or model entities are introduced.

free parameters (4)
  • FLORES-200 token premiums r_{m,L} = Qwen de 1.56 / th 2.55 / sw 1.94; Llama de 1.58 / th 2.19 / sw 1.93
    Measured from FLORES-200 devtest with each model's own tokenizer (§3). They are external measurements, not fitted to the target gap, but the central estimand Δ_L(B)=accN(⌊rB⌋)−accN(B) depends on them, and the paper's normalizer sensitivity analysis shows one cell is sensitive to this choice.
  • Tail-conservatism factor = 1.3
    Pre-specified multiplier applied to raw confirmatory p-values before Holm correction, chosen because calibration with 250 item clusters found mild anti-conservatism in the α/6 tail (Appendix E). It does not fit the estimand, but it is a hand-chosen statistical constant.
  • SESOI margin and equivalence margin = 5 points
    Pre-specified boundary for what counts as a meaningful gap artifact in H1-SESOI, the independent peak tests, and the B*=1024 equivalence tests. Chosen by hand.
  • Instrument validity gate = 30% median-length reduction
    Pre-specified gate for the announcement experiment: cells that do not reduce median output length by at least 30% are deemed uninformative. Chosen by hand in Appendix I.
assumptions (4)
  • domain assumption FLORES-200 token premium r_{m,L} is an appropriate length normalizer for MGSM reasoning traces
    Entered in §3, Equation (1), where the premium-scaled budget ⌊rB⌋ defines the length-matched comparison. Appendix D shows the behavioral trace-length ratio differs and that the Qwen Swahili 5-point artifact is sensitive to this choice.
  • domain assumption Prefix scoring a stored 4096-token generation equals hard-capped decoding at every budget B
    The ledger evaluates the length-B prefix of a 4096-cap generation (§3). The announcement experiment (§4.4) shows accuracy is not a function of the enforced cap alone, so this assumption is only partially tested and only for the six pre-specified confirmatory cells.
  • domain assumption Correctness is approximately absorbing for the correct-emission sub-CDF identity
    Appendix J uses G(t)=P(C=1,E≤t) and the identity Δ_L(B)=G(⌊rB⌋)−G(B). Genuine answer revision occurs in 1.35% of records and correct-to-wrong revision in 0.52%, so absorption is approximate; the paper argues the bias is small.
  • domain assumption Automated GlotLID language labels reliably separate NATIVE and TRANSLATE-ACT reasoning
    Used in §5 for the trace-language audit. The frozen human validation is outstanding, and the Swahili macrolanguage mapping was adopted after inspecting a failed criterion, which the authors treat as a post-hoc analytic decision.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap." pith.science (2026). https://pith.science/paper/TAYZGEGC

@misc{pith2026260804160,
  author       = {Pith},
  title        = {Pith review of: Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TAYZGEGC}},
  note         = {Machine review of arXiv:2608.04160}
}
abstract

Multilingual evaluations report accuracy at a single output-token cap, but languages need different numbers of tokens to express the same content, so the cap is a hidden experimental variable. We test whether the native-vs-translate gap on MGSM (German, Thai, Swahili) is a token-budget artifact for Qwen3-8B and Llama-3.1-8B-Instruct under four prompting strategies. The measured gap swings by up to 57 points across budgets, length normalization moves it by up to 38.9 points where the cap binds, and at tight caps normalization can reverse which strategy scores higher. We prospectively froze the sweep's three Qwen peaks and its near-zero value at 1024 and evaluated them on 540,000 independently hard-capped decodes: a second frozen family of six Holm-corrected tests rejects every null. The frozen test at $B^*=1024$ still fails to reject because native accuracy has already saturated there; above saturation, the residual difference is a strategy-performance gap, not an identified reasoning deficit. The same truncation channel prices a cost-ordered adaptation ladder: a cross-fitted Thai vocabulary extension closes 0.0 points of the gap at the frozen budget and 4.9 points where 19% of traces still truncate. A third frozen family varies only the announced budget at a fixed enforced cap; announcing 128 rather than 2048 tokens moves Thai native accuracy by 5.1 points, so accuracy is not a function of the enforced cap alone. A correct-emission timing identity computed from one long-cap run matches the three pre-specified MGSM peaks to 0.65 points and, in an exploratory Qwen-only analysis of three further benchmarks, tracks held-out items to 0.92 points, locating the peak exactly in five of seven cells. Treat the output cap as an independent variable and report accuracy across the budget regime, not at a single budget.

Figures

Figures reproduced from arXiv: 2608.04160 by the authors.

Figure 1
Figure 1. Qwen NATIVE accuracy by budget. Shaded intervals mark each language’s peak normalization interval [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. The measured gap gap(B) = accT (B) − accN (B) by output budget, token-frame prefix accuracies from the stored ledger. Negative values at tight caps are the crossovers of §4.3; the plateaus at large B are the saturated strategy-performance gaps of §4.1. The Qwen Thai gap peaks near B = 384, where TRANSLATE-ACT has largely saturated while NATIVE is still emitting answers. TIVE output-length p90 is 4096 and 25.1% never… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 16 canonical work pages

  1. [4]

    Transactions of the Association for Computational Linguistics (TACL) , volume =

    The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation , author =. Transactions of the Association for Computational Linguistics (TACL) , volume =

    work page

  2. [5]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =

    Kargaran, Amir Hossein and Imani, Ayyoob and Yvon, Fran. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =

    work page 2023

  3. [6]

    Proceedings of the Seventh Conference on Machine Translation (WMT) , pages =

    Rei, Ricardo and de Souza, Jos. Proceedings of the Seventh Conference on Machine Translation (WMT) , pages =

  4. [7]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models , author =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    work page 2023

  5. [8]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    Language Model Tokenizers Introduce Unfairness Between Languages , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =

    work page

  6. [11]

    Broken Chains: The Cost of Incomplete Reasoning in

    Su, Ian and Purushothaman, Gaurav and Narayan, Jey and Goel, Ruhika and Zhu, Kevin and Dev, Sunishchal and More, Yash and Chaudhary, Maheep , journal =. Broken Chains: The Cost of Incomplete Reasoning in. 2026 , eprint =

    work page 2026

  7. [14]

    2025 , eprint =

    Luo, Wenyang and Zhao, Wayne Xin and Sha, Jing and Wang, Shijin and Wen, Ji-Rong , journal =. 2025 , eprint =

    work page 2025

  8. [19]

    Continual Pre-Training for Cross-Lingual

    Fujii, Kazuki and Nakamura, Taishi and Loem, Mengsay and Iida, Hiroki and Ohi, Masanari and Hattori, Kakeru and Shota, Hirai and Mizuki, Sakae and Yokota, Rio and Okazaki, Naoaki , journal =. Continual Pre-Training for Cross-Lingual. 2024 , eprint =

    work page 2024

Show all 29 references
  1. [20]

    2023 , doi =

    Dobler, Konstantin and de Melo, Gerard , booktitle =. 2023 , doi =

    2023

  2. [21]

    Mortensen, Noah A

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural La...

    2023

  3. [22]

    Yiming Cui, Ziqing Yang, and Xin Yao. 2023. https://arxiv.org/abs/2304.08177 Efficient and effective text encoding for chinese llama and alpaca . arXiv preprint arXiv:2304.08177

    2023 arXiv

  4. [23]

    Konstantin Dobler and Gerard de Melo. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.829 FOCUS : Effective embedding initialization for monolingual specialization of multilingual models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process...

    2023 doi

  5. [24]

    Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. https://arxiv.org/abs/2404.17790 Continual pre-training for cross-lingual LLM adaptation: Enhancing japanese language capabi...

    2024 arXiv

  6. [25]

    Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzm \'a n, and Angela Fan. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of t...

    2022

  7. [26]

    Aaron Grattafiori et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . arXiv preprint arXiv:2407.21783

    2024 arXiv

  8. [27]

    Amir Hossein Kargaran, Ayyoob Imani, Fran c ois Yvon, and Hinrich Sch \"u tze. 2023. GlotLID : Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155--6218

    2023

  9. [28]

    Hyunwoo Ko, Guijin Son, and Dasol Choi. 2025. https://arxiv.org/abs/2501.02448 Understand, solve and translate: Bridging the multilingual mathematical reasoning gap . arXiv preprint arXiv:2501.02448

    2025 arXiv

  10. [29]

    Maxence Lasbordes, Am \'e lie Chatelain, and Djam \'e Seddah. 2026. https://arxiv.org/abs/2605.26735 Rethinking the multilingual reasoning gap with layer swap . arXiv preprint arXiv:2605.26735

    2026 arXiv

  11. [30]

    Hedderich, and Hinrich Sch \"u tze

    Yihong Liu, Raoyuan Zhao, Michael A. Hedderich, and Hinrich Sch \"u tze. 2026. https://arxiv.org/abs/2605.09548 Crosslingual on-policy self-distillation for multilingual reasoning . arXiv preprint arXiv:2605.09548

    2026 arXiv

  12. [31]

    Wenyang Luo, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. 2025. https://arxiv.org/abs/2505.19126 MMATH : A multilingual benchmark for mathematical reasoning . arXiv preprint arXiv:2505.19126

    2025 arXiv

  13. [32]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. 2025. https://arxiv.org/abs/2501.19393 s1: Simple test-time scaling . arXiv preprint arXiv:2501.19393

    2025 arXiv

  14. [33]

    Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al

    NLLB Team , Marta R. Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. https://arxiv.org/abs/2207.04672 No language left behind: Scaling human-centered machine tran...

    2022 arXiv

  15. [34]

    Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. In Advances in Neural Information Processing Systems (NeurIPS), volume 36

    2023

  16. [35]

    Qwen Team . 2025. https://arxiv.org/abs/2505.09388 Qwen3 technical report . arXiv preprint arXiv:2505.09388

    2025 arXiv

  17. [36]

    de Souza, Duarte Alves, Chrysoula Zerva, Ana C

    Ricardo Rei, Jos \'e G.C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F.T. Martins. 2022. COMET-22 : Unbabel- IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Mac...

    2022

  18. [37]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. https://arxiv.org/abs/2210.03057 Language models are multilingual chain-of-thought reasoners . In The...

    2023 arXiv

  19. [38]

    Gianluigi Silvestri and Edoardo Cetin. 2026. https://arxiv.org/abs/2603.13274 Learning from partial chain-of-thought via truncated-reasoning self-distillation . arXiv preprint arXiv:2603.13274

    2026

  20. [39]

    Ian Su, Gaurav Purushothaman, Jey Narayan, Ruhika Goel, Kevin Zhu, Sunishchal Dev, Yash More, and Maheep Chaudhary. 2026. https://arxiv.org/abs/2602.14444 Broken chains: The cost of incomplete reasoning in LLM s . arXiv preprint arXiv:2602.14444

    2026

  21. [40]

    Hedderich, Hinrich Sch \"u tze, and Yihong Liu

    Jiaqiao Zhang, Zhoujun Li, Raoyuan Zhao, Jian Lan, Thomas Seidl, Michael A. Hedderich, Hinrich Sch \"u tze, and Yihong Liu. 2026. https://arxiv.org/abs/2605.27715 Beyond input understanding: Diagnosing multilingual mathematical reasoning with directed acyclic trace graphs . ar...

    2026 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.