{"id":"5c3d021c-9c25-4a1b-84db-873074bc7672","arxiv_id":"2505.05408","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Test-time scaling of English-centric reasoning models improves multilingual math accuracy for models at 3B parameters and above, with English-heavy 'quote-and-think' reasoning, but gains do not transfer to cultural common-sense domains.","lead":"This paper tests whether giving multilingual questions to English-trained reasoning models, and letting them think longer, improves math performance across languages. It finds clear gains for models of 3B parameters and larger, and identifies a 'quote-and-think' pattern in how they reason about non-English input.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 'outperform models twice their size' is not established because the larger baselines in Table 1 were not given the same test-time budget; the Pareto frontier in Figure 1b omits them.","rationale":"The reader's weakest_assumption concerns the 3B threshold and the attribution of prior negative results to model size. I agree that comparison is confounded by differences in base model, finetuning data scale, and recipe, but that threshold is a secondary explanatory claim about prior work, not the paper's headline result. The more load-bearing issue for the central claim is the asymmetric test-time budget in the 'outperform models twice their size' comparison in Table 1. The paper frames this as a headline in the abstract and uses it to argue that test-time scaling of a light recipe is a strong multilingual baseline. The authors' own Pareto analysis is the right tool for a cost-aware comparison, but it omits the larger baselines, so a reader cannot tell whether R1-Distill-Qwen-32B or Gemma-3-27B-it would also improve with a larger token budget and overtake s1-14B. The internal scaling result is well-supported: accuracy rises with maximum thinking tokens for s1-3B and above, and the paper reports token counts and baselines carefully. The quote-and-think analysis is qualitative but plausible, and the cross-domain results honestly document overthinking and limited generalization. I am not objecting to the core finding; I am objecting to one headline comparison that is currently not a controlled experiment. The reader's CONDITIONAL verdict already captures the need for caution, so my concern does not move the verdict, but it identifies a more direct target for the required control.","tokens_in":29827,"tokens_out":7405,"duration_ms":79879,"concrete_test":"Re-run Table 1 with R1-Distill-Qwen-32B and Gemma-3-27B-it under identical budget forcing: same maximum thinking tokens (0.5k, 1k, 2k, 4k, 8k), same truncation and extrapolation ('Wait') protocols, same greedy decoding, and the same lm-evaluation-harness on MGSM. Plot accuracy against inference FLOPs (2 * N * D_inference) together with s1-14B. If at 8k tokens or at matched FLOPs either larger model reaches or exceeds s1-14B's 81.0% average, the 'outperform models twice its size' claim fails; if they remain below at matched compute, the claim is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim that s1-14B 'outperforms models twice its size' rests on Table 1, where s1-14B is run with an 8k maximum thinking-token budget via extrapolation (average generation length 2352 tokens), while R1-Distill-Qwen-32B and Gemma-3-27B-it are evaluated 'without any inference budget constraint' (average generation lengths 1354 and 462 tokens, respectively). The abstract states this as a headline result, and the practical recommendation that a lightweight English-only finetuning plus test-time token scaling is a strong multilingual baseline depends on it. The paper's own Pareto-frontier analysis in Figure 1b covers only s1 and Qwen models, so it never shows where the larger baselines sit on the accuracy-FLOPs frontier under test-time scaling. Unless those baselines are also given the opportunity to spend comparable inference compute, for example by applying the same 'Wait' extrapolation or a matching token cap, the reported gap may reflect asymmetric compute budgets rather than any superiority of the lightweight recipe. This is not an accusation of misreporting; it is an incomplete control. The internal s1 scaling trend in Figure 1a is credible, but the 'outperform twice-size models' component of the central claim is under-supported as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether test-time scaling of English-centric reasoning language models (RLMs), specifically the s1 family built from Qwen2.5-Instruct, transfers to multilingual mathematical reasoning. Using MGSM across 11 languages, the authors report that increasing the maximum thinking-token budget improves accuracy for s1 models of 3B parameters and above, with the largest gains at 14B, and that s1-14B outperforms several larger baselines including R1-Distill-Qwen-32B and Gemma-3-27B-it. The paper also documents a 'quote-and-think' language-mixing pattern in CoTs, studies methods for forcing reasoning into non-English languages, finds that reasoning in high-resource languages is more accurate and token-efficient than in low-resource languages, and reports limited out-of-domain generalization on Global-MMLU, FORK, and COPAL-ID. The authors conclude that data-efficient English reasoning finetuning plus test-time token scaling is a strong multilingual baseline, while cautioning about low-resource languages and out-of-domain tasks.","tokens_in":30040,"tokens_out":4296,"duration_ms":44263,"significance":"If the central claims hold, the paper provides a practical and low-cost recipe for multilingual math reasoning: English-only finetuning on 1k samples followed by test-time token scaling, and it adds evidence against the assumption that reasoning abilities are inherently English-bound. The quote-and-think analysis is a plausible and mechanistically interesting account of how English-centric RLMs parse non-English inputs, and the language-forcing experiments are a systematic contribution. The paper also includes useful cautionary results on overthinking and out-of-domain degradation. Strengths include the explicit train-test overlap check, the open release of code and models, and the breadth of ablations across model sizes and inference budgets. The main weakness is that the headline comparison to 'models twice its size' does not control for inference compute, and the attribution of prior negative results to model size alone rests on comparisons that confound multiple factors.","major_comments":[{"comment":"The claim that s1-14B 'outperforms models twice its size' is not supported as stated because the comparison is not controlled for inference compute. In Table 1, s1-14B uses extrapolation with an 8k maximum thinking-token budget and has an average generation length of 2352.3 tokens, while R1-Distill-Qwen-32B and Gemma-3-27B-it are evaluated 'without any inference budget constraint' and have average lengths of 1353.8 and 461.7 tokens, respectively. The Pareto frontier in Figure 1b covers only s1 and Qwen models, so it does not show where the larger baselines would sit if given comparable test-time compute. The observed gap could be partly or wholly due to asymmetric compute budgets rather than any superiority of the lightweight recipe. The authors should apply the same budget-forcing procedure (e.g., the same 'Wait' extrapolation or an 8k token cap) to the larger baselines and report the resulting accuracies, or qualify the claim accordingly.","section":"Table 1 and Section 4.2"},{"comment":"The conclusion that crosslingual test-time scaling requires at least 3B parameters, and that the negative findings of [17] are 'due to their usage of 1.5B models,' is under-supported because the comparison confounds parameter count with model family, base model, training-data scale, and recipe. The s1-1.5B model and the R1-Distill-Qwen-1.5B model used in [17] differ in base model, in finetuning data size (1k vs. 800k samples), and in training procedure. The observed 3B threshold is therefore established only for the s1 family on Qwen2.5 bases. The authors should either test additional RLM families at 1.5B and 3B, or explicitly limit the threshold claim to s1 models.","section":"Section 4.1 and Section 8"}],"minor_comments":[{"comment":"There is a typo in 'two main obsevations' (should be 'observations').","section":"Section 4.1"},{"comment":"The sentence ending 'lm-evaluation-harness library [47] as the main evaluation framework.3.' contains a stray '3.' that should be removed.","section":"Section 3"},{"comment":"The caption says 'CoTs prdocued' (should be 'produced').","section":"Box 2"},{"comment":"The sentence 'we force the model to all M possible languages' is grammatically incomplete; it should read 'we force the model to reason in all M possible languages.'","section":"Section 6.3"},{"comment":"The reported correlation of -0.811 between token count and accuracy is based on only 11 language-level aggregate points; a confidence interval or a per-item analysis would make the strength of the claim clearer.","section":"Figure 5"},{"comment":"The sentence listing prior work, 'all prior state-of-the-art models that involve finetuning on multilingual data such as MetaMath, MAPO, and [52, 37, 49, 35]', is awkwardly phrased and the citation [52] repeats the MetaMath reference; consider rewriting for clarity.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is empirically solid on its internal scaling trend and the language-mixing analysis is a valuable contribution. The main risk is the headline claim about outperforming twice-size models, which is currently based on an asymmetric compute comparison; this is fixable with a matched-budget control. The comparison with [17] should also be softened to avoid the implication of a general 3B threshold. If the authors add the missing controls, the paper would be a strong candidate for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's core finding — that giving English-centric s1 models more thinking tokens improves multilingual math for 3B and up — is credible and well supported by Figure 1a and the language-by-language tables. The quote-and-think analysis is genuinely new and useful; it gives a concrete mechanism for how an English-trained reasoner can parse and reason about non-English inputs. The language-forcing experiments are also a solid practical contribution, especially the token-efficiency result for high-resource languages and the correlation in Figure 5. The out-of-domain limits are honestly reported.\n\nWhere the paper overreaches is the headline claim that s1-14B 'outperforms models twice its size.' The stress-test note is right: R1-Distill-Qwen-32B and Gemma-3-27B-it were run without any inference budget constraint, while s1-14B gets the 8k max-token extrapolation treatment. Their average generation lengths (1354 and 462 tokens) are far below s1-14B's 2352. It is entirely possible that those bigger models would close or reverse the gap if given the same opportunity to spend test-time compute. The paper's own Pareto frontier in Figure 1b only plots s1 and Qwen, so it never shows where the bigger baselines sit. This is an incomplete control, not fraud, but it means the abstract's most eye-catching sentence is not established.\n\nThe claim that Son et al.'s negative result comes from model size alone is also under-supported. s1-1.5B and DeepSeek-R1-1.5B differ in base model, training data scale (1k vs 800k samples), and recipe; the paper does not isolate parameter count. The 3B threshold is an observation about one model family, not a tested law. To their credit, the authors hedge in the discussion with 'corroborates,' but the Section 4.1 language is stronger than the evidence.\n\nMinor stuff: no commit hash in the repo link, and the GlobalMMLU parse relies on GPT-4o-mini. Both are fixable and not load-bearing.\n\nWho should read this: anyone working on multilingual reasoning, test-time scaling, or low-resource math evaluation. The recipe (1k English samples plus token scaling) is actionable even if the size-comparison claim needs revision.\n\nRecommendation: send to peer review. The internal scaling result and the linguistic analysis are worth referee time. Require the authors to either give the larger baselines a comparable test-time budget or drop the 'twice its size' claim from the abstract. The 3B threshold discussion should be reworded as a hypothesis about model families, not a general law.","headline":"The internal scaling result is real and useful, but the 'outperform models twice its size' headline is not yet supported because the big baselines never got the same test-time budget.","tokens_in":30641,"tokens_out":1459,"would_cite":true,"duration_ms":17485,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Scaling thinking tokens transfers English math reasoning to ten languages.","keywords":["test-time scaling","multilingual reasoning","chain-of-thought","crosslingual generalization","low-resource languages","language forcing","MGSM","reasoning language models"],"falsifier":"Take a 1.5B and a 3B model from the same base family, finetune both on identical 1k English long-CoT samples, and measure MGSM accuracy at 500 versus 8,000 maximum thinking tokens; if the 1.5B model still fails to gain while the 3B model gains, the capacity threshold is confirmed, and if the 1.5B model gains too, the threshold is an artifact of the specific model comparison.","tokens_in":29610,"feed_emoji":"🧠","tokens_out":5517,"duration_ms":53974,"temperature":0.7,"pith_summary":"This paper asks whether an English-centric reasoning model, one finetuned on a thousand English math problems with long chain-of-thought traces, can reason in other languages if it is simply given more thinking tokens at inference time. It finds that test-time scaling works across all ten MGSM languages for models of 3B parameters and above, with the 14B model outperforming models twice its size, such as DeepSeek-R1-Distill-Qwen-32B, on multilingual math. The gains are not just translation effects: the model's chain-of-thought stays mostly English but repeatedly quotes the non-English prompt and interprets it, a pattern the paper calls quote-and-think. Forcing the model to think in a low-resource language hurts accuracy and requires far more tokens, while thinking in high-resource languages is nearly as good as English and more efficient. The paper concludes that crosslingual test-time scaling is a viable lightweight baseline, but out-of-domain and low-resource reasoning remain unsolved.","feed_headline":"More thinking tokens lift math accuracy across 10 languages","feed_subtitle":"English-only finetuning plus longer chain-of-thought beats twice-larger rivals on multilingual math, the paper shows.","key_machinery":"Budget forcing on long chain-of-thoughts: at each test, the model generates until it hits a maximum thinking-token cap, called truncation, or receives an appended 'Wait' token to continue reasoning, called extrapolation. This is the lever that converts extra inference FLOPs into accuracy. The complementary mechanism is quote-and-think, the dominant language-mixing pattern in which the model's mostly English reasoning quotes non-English phrases from the prompt and interprets them, which is what lets an English-trained reasoner parse non-English math problems.","core_discovery":"The central claim is that English-only reasoning finetuning transfers across languages through test-time compute: increasing the maximum number of thinking tokens from 500 to 8,000 improves average MGSM accuracy for s1 models of 3B, 7B, 14B, and 32B parameters, including low-resource languages like Swahili and Telugu, and a 14B s1 model reaches 81.0% average accuracy, out-scoring DeepSeek-R1-Distill-Qwen-32B at 72.7% and Gemma-3-27B-it at 79.8%. The 1.5B model only gains 1.8 points, which the authors attribute to a capacity threshold rather than to a failure of test-time scaling. The paper also shows that the model's multilingual reasoning flows through a quote-and-think mechanism: English chains-of-thought lift non-English phrases from the question into quotation marks and reason about their meaning, a behavior present in 68.3% of the s1 training samples. When the reasoning language is forced, high-resource languages such as English, French, and German perform near English, while low-resource reasoning languages like Swahili and Telugu drop accuracy and need roughly 3.5 times more compute.","pith_inferences":["If the 3B threshold is a general property of English-centric reasoning models, test-time scaling could make small multilingual models competitive without retraining, but the comparison does not isolate parameter count because s1 and DeepSeek-R1-1.5B differ in base model, training data scale, and recipe.","The quote-and-think pattern suggests an alternative to translation-based multilingual pipelines: instead of translating inputs, one could explicitly train or prompt models to quote and reinterpret the original phrase, preserving syntactic and cultural cues that translation would discard.","The negative correlation between token count and accuracy across reasoning languages implies that tokenizer fairness, not just model knowledge, shapes which languages are affordable to reason in, so better tokenizers could shift the Pareto frontier for low-resource languages.","A testable extension is to combine translated 'Wait' forcing with English-heavy reasoning for low-resource queries, letting English do the bulk of the computation while retaining the user's language in the output.","The paper does not test whether the same crosslingual scaling holds for non-mathematical STEM tasks such as code or formal proofs, where the quote-and-think pattern may behave differently."],"forward_implications":["Practitioners can use test-time scaling of English-centric reasoning models as a strong multilingual math baseline without any multilingual finetuning.","Data-efficient English finetuning with about 1,000 samples preserves multilingual ability better than large-scale distillation, reducing catastrophic forgetting for low-resource languages.","English-centric reasoning models should be allowed to reason in high-resource languages rather than forced into the query language, since in-language forcing in low-resource languages costs both accuracy and tokens.","Test-time scaling does not generalize to non-STEM domains; on cultural commonsense benchmarks, more thinking tokens can even reduce accuracy through overthinking.","A 3B-parameter threshold separates models that benefit from crosslingual test-time scaling from those that do not, contradicting earlier claims that much larger models would be needed."],"supporting_citations":[{"why":"Supplies the s1 models, the 1k English reasoning samples, and the budget forcing method that the whole study depends on.","marker":"[9]"},{"why":"Supplies the MGSM benchmark, the ten languages, and the few-shot CoT baselines used for evaluation.","marker":"[31]"},{"why":"Identifies Qwen2.5-Instruct as the multilingual base models for s1 and as the primary zero-shot and few-shot baselines.","marker":"[16]"},{"why":"Provides DeepSeek-R1 as the source of distilled reasoning data and defines the R1-Distill-Qwen baselines that s1 outperforms.","marker":"[5]"},{"why":"Is the contemporaneous negative result on 1.5B models that this paper directly contrasts with its 3B threshold finding.","marker":"[17]"},{"why":"Introduces the test-time scaling paradigm and the FLOPs approximation used for the Pareto frontier analysis.","marker":"[1]"},{"why":"Supplies Global-MMLU, the multilingual cross-domain benchmark used to test STEM and non-STEM generalization.","marker":"[41]"},{"why":"Supplies the FORK cultural commonsense benchmark where more thinking tokens reduce accuracy.","marker":"[43]"},{"why":"Supplies COPAL-ID, the Indonesian cultural causal reasoning benchmark used for out-of-domain evaluation.","marker":"[45]"}],"fun_headline_variants":["More thinking tokens bridge language gaps for LLM math","Scaling compute lets English-trained LLMs ace multilingual math","Test-time scaling: key to crosslingual reasoning in models","Quote-and-think: English CoTs handle non-English math","Longer English CoTs outscore models twice the size on MGSM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the s1 model family represents English-centric reasoning models generally, so the observed 3B-parameter threshold is attributed to parameter count rather than to differences in base model, finetuning data scale, or training recipe between s1 and the 1.5B model it is compared against.","fun_headline_variants_meta":{"raw":{"variants":["More thinking tokens bridge language gaps for LLM math","Scaling compute lets English-trained LLMs ace multilingual math","Test-time scaling: key to crosslingual reasoning in models","Quote-and-think: English CoTs handle non-English math","Longer English CoTs outscore models twice the size on MGSM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1686,"prompt_tokens":1023,"completion_tokens":663,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":577}},"tokens_in":639,"tokens_out":663,"duration_ms":7256,"temperature":1.0,"reasoning_tokens":577,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:04:11.884734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a 1.5B and a 3B model from the same base family, finetune both on identical 1k English long-CoT samples, and measure MGSM accuracy at 500 versus 8,000 maximum thinking tokens; if the 1.5B model still fails to gain while the 3B model gains, the capacity threshold is confirmed, and if the 1.5B model gains too, the threshold is an artifact of the specific model comparison.","supporting_citations":[{"cited_title":"Language models are multilingual chain-of-thought reasoners","cited_arxiv_id":null,"evidence_quote":"Supplies the MGSM benchmark, the ten languages, and the few-shot CoT baselines used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Global-MMLU, the multilingual cross-domain benchmark used to test STEM and non-STEM generalization."},{"cited_title":"FORK: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models","cited_arxiv_id":null,"evidence_quote":"Supplies the FORK cultural commonsense benchmark where more thinking tokens reduce accuracy."},{"cited_title":"COPAL- ID: Indonesian language reasoning with local culture and nuances","cited_arxiv_id":null,"evidence_quote":"Supplies COPAL-ID, the Indonesian cultural causal reasoning benchmark used for out-of-domain evaluation."}],"review_version":1}