{"id":"f7d754e5-e549-4e4d-9d25-20c24d66f56f","arxiv_id":"2412.12609","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Training one model on Program-of-Thought solutions in four programming languages raises math accuracy for each language, and answer mixing outperforms single-language augmented training by up to about 6 percentage points.","lead":"This paper fine-tunes small language models on math problems solved in Python, C++, Java, and Matlab, then mixes the answers. The authors report accuracy gains of about 2.5 points per language and up to 6 points over single-language training when a language selector is used.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 6% hybrid gain is not isolated to multilingualism: the best posterior method adds a Llama3-8B scorer that SinglePoT-DA.SC lacks, and no same-scorer single-language control is reported.","rationale":"Reader's weakest_assumption (diversity) is valid and should be fixed, but I find a second confound more decisive for the headline. The posterior hybrid that delivers the largest gains uses an external Llama3-8B-Instruct model as scorer, while the single-language baseline uses only self-consistency voting. This changes both the candidate pool and the selection mechanism, so the comparison is not an ablation of multilingualism. The MultiLingPoT self-consistency result, which does not use the external scorer, is only ~0.4 points above SinglePoT-DA.SC (64.29 vs 63.88 simple; 37.78 vs 37.45 complex), suggesting that most of the advertised 6% comes from the scorer step. The proposed test — applying the same Llama3 scorer to single-language Python candidates — would isolate the contribution. If the scorer closes the gap, the 'multilingual' claim reduces to a diversity/augmentation effect plus a stronger selector; if not, the paper's conditional acceptance with these ablations would be justified. Hence verdict remains conditional, and I agree partly with the reader's diversity concern but would add the scorer control as the primary requirement.","tokens_in":16848,"tokens_out":8176,"duration_ms":73952,"concrete_test":"Run SinglePoT-DA.SC with the same aggregation module: sample K Python solutions per question from the SinglePoT-DA model (K = number of MultiLingPoT candidates, e.g., 4 or 8), and have the same Llama3-8B-Instruct scorer select the final answer. Evaluate on the same GSM8K/MATH splits. If the single-language + same-scorer accuracy reaches 66.42/40.53, the headline gain is from the scorer, not from multilingualism; if it remains near 63.88/37.45, multilingual candidates are the active ingredient. Repeat both arms with 3 seeds for statistical comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison for the headline 6% claim is between MultiLingPoT posterior Llama3 Scorer (Tables 4/5) and SinglePoT-DA.SC. These differ in two ways: candidate answers are generated in four languages vs one, and selection is performed by Llama3-8B-Instruct vs plain voting. The paper reports no control where the same Llama3-8B-Instruct scorer selects among K Python samples from SinglePoT-DA. Without that control, the +2.5/+3.1 average advantage (66.42 vs 63.88 simple; 40.53 vs 37.45 complex) cannot be assigned to multilingual training; it may come from the stronger aggregation model. The diversity confound in Section 5.2 similarly prevents attributing the per-language training gains to language diversity rather than generic solution diversity, but the missing scorer control is the more direct threat to the quantitative 6% claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MultiLingPoT, a multilingual Program-of-Thought method that fine-tunes a code language model on solutions written in Python, C++, Java, and Matlab for GSM8K and MATH. It introduces a new multilingual dataset generated by ChatGPT and filtered by execution, and explores several prior and posterior hybrid strategies (case-based choice, small-model/LLM scorers, DPO, self-consistency, and voting-plus-scoring) to select the best language or answer for each problem. Experiments on CodeLlama-7B, DeepSeekCoder-7B, CodeLlama-Python-7B, and Llama3-8B report that MultiLingPoT training improves each language's accuracy by about 2.5% on average and that the best posterior hybrid outperforms single-language PoT with data augmentation by about 6%.","tokens_in":17002,"tokens_out":3865,"duration_ms":34405,"significance":"If the reported gains are robust, the paper provides a useful recipe and a new resource for multilingual program-based mathematical reasoning, and it explicitly addresses a gap in the existing PoT literature. The construction of a filtered multilingual PoT dataset and the systematic comparison of several mixing strategies are valuable contributions. The paper also tests across multiple base models and out-of-domain datasets, which strengthens the empirical scope. However, the central quantitative claims currently rest on comparisons that are confounded by data diversity and by the choice of answer-selection mechanism, and there are no variance estimates; these issues need to be resolved before the specific 2.5% and 6% numbers can be accepted.","major_comments":[{"comment":"The comparison between MultiLingPoT's individual languages and SinglePoT-DA is confounded by data diversity. The paper itself attributes the gains to \"solutions in different programming languages offer more diversity,\" but SinglePoT-DA repeats identical Python solutions multiple times. To support the claim that multilingualism, rather than generic solution diversity, is responsible for the improvement, a control with diverse single-language augmentation (for example, multiple distinct Python solution programs per problem) is needed.","section":"Section 5.2, Tables 2 and 3"},{"comment":"The headline 6% improvement compares MultiLingPoT posterior Llama3 Scorer (66.42 simple, 39.03 complex averages) with SinglePoT-DA.SC (63.88 simple, 37.45 complex averages). These systems differ in two ways: the training data are multilingual versus single-language, and the answer selection is performed by a Llama3-8B-Instruct scorer versus simple majority voting. Without a control in which the same Llama3-8B-Instruct scorer selects among multiple Python samples from SinglePoT-DA, the 6% gain cannot be attributed to multilingual training; it may reflect the stronger aggregation model. Please add this control or soften the causal claim.","section":"Abstract and Section 5.3, Tables 4 and 5"},{"comment":"All reported accuracies come from single runs without error bars or multiple seeds. Several per-language differences are small (for example, C++ on the simple GSM8K training set is 64.97 in both SinglePoT and MultiLingPoT, Table 2; in Table 6, CodeLlama-Python's Python language row is 33.71 versus SinglePoT Python-DA 35.32), so the 2.5% average improvement could be within training noise. At least two seeds with standard deviations, or bootstrap confidence intervals, are needed to support the quantitative claims.","section":"Section 5.2 and Tables 2-6"},{"comment":"The paper criticizes Luo et al. (2024) for prompt-based multilingual PoT and for a biased comparison, but it does not include that method as an experimental baseline anywhere in the evaluation. Since Luo et al. is the closest prior work on multilingual PoT, a direct comparison is necessary to substantiate the claim that the proposed SFT-based approach is preferable for small models.","section":"Section 2 and Section 5.1"}],"minor_comments":[{"comment":"There is a duplicated word in \"The posterior hybrid strategy is is a hybrid strategy\".","section":"Section 4.2"},{"comment":"There are spelling errors: \"Basides\" should be \"Besides\" and \"comparsion\" should be \"comparison\".","section":"Section 2"},{"comment":"\"a prior hybrid method\" should be \"a prior hybrid method\" or \"a priori hybrid method\", and the phrase has a typographical spacing issue.","section":"Limitations"},{"comment":"The table headers \"Count Int. Num.\" and the term \"V oting\" contain odd spacing; please use standard formatting.","section":"Tables 4 and 5"},{"comment":"The conclusion that prior hybrid strategies help the model select the appropriate language is overstated for the simple datasets: in Table 4, most prior methods perform near or below the Random baseline (e.g., Llama3 Scorer 58.76 versus Random 59.56). The text should acknowledge this more explicitly.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's resource release and broad evaluation are positive, but the central quantitative claims are currently under-supported by the experimental design. The missing same-scorer single-language control and the lack of variance estimates are the main blockers. I would consider acceptance after these controls are added and the claims are adjusted accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable and clearly written empirical study of multilingual Program-of-Thought via supervised fine-tuning. It contributes a new four-language dataset, a sensible prior/posterior taxonomy of selection strategies, and a check on three model families. The headline claim — about 6% over single-language PoT with data augmentation — is not yet supported, because the best posterior method swaps in a Llama3-8B reranker that the baseline lacks.\n\nNew here: prior work on multilingual PoT (Luo et al., 2024) was prompting-based. This paper does SFT, which is a natural extension, and it explores language selection more thoroughly: Case-Based Choice, trained scorers (Bert, CodeBert, Llama3), DPO, self-consistency, and voting+scoring. The dataset is built via ChatGPT and filtered by execution, and it is released. The paper also evaluates on DeepseekCoder, CodeLlama-Python, and Llama3, which gives some generality. That is a useful empirical package for small-model math reasoning.\n\nWhere it goes soft: the 6% claim compares MultiLingPoT posterior Llama3 Scorer against SinglePoT-DA.SC. The two differ in two ways at once: the candidate solutions come from four languages instead of one, and the final answer is chosen by an 8B LLM reranker instead of majority voting. There is no control where that same Llama3 reranker picks among K Python-only candidates. Without it, the +2.5 to +3.1 average gains could be mostly a selector effect. The diversity confound noted in the paper's Section 5.2 is secondary but real: a diverse single-language augmentation control would be needed to attribute the gains to multilingualism specifically. On top of that, the per-language \"2.5% improvement\" in Section 5.2 is computed against SinglePoT models trained on much less data. Against the data-matched SinglePoT-DA baseline, the per-language differences mostly vanish, so the \"languages learn from each other\" claim is not currently supported. Also no error bars and a single seed.\n\nWho this is for: researchers working on math reasoning with small code models. The dataset and the taxonomy of mixing strategies are the most valuable parts; the quantitative conclusions need tightening.\n\nRecommendation: yes, send to peer review. The concerns are addressable with additional control experiments (same reranker on Python-only candidates, a diverse single-language augmentation control, multiple seeds). The paper is serious and would benefit from a referee round.","headline":"Genuine SFT-based multilingual PoT with a usable dataset, but the headline 6% gain is confounded by a stronger answer selector; the per-language training gains are mostly a data-volume effect.","tokens_in":17547,"tokens_out":4709,"would_cite":true,"duration_ms":40285,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mixing four programming languages lifts math AI by 6 percent","keywords":["mathematical reasoning","program-of-thought","multilingual code generation","supervised fine-tuning","code language models","self-consistency","language selection","data augmentation"],"falsifier":"Build a diverse single-language control: for each GSM8K and MATH training problem, collect several distinct correct Python solutions with different algorithms, code styles, or paraphrases so the Python-only training set matches MultiLingPoT's sample count and diversity. If that model matches or beats the 6% advantage, then multilingualism itself is not the causal driver.","tokens_in":16647,"feed_emoji":"🧮","tokens_out":8398,"duration_ms":70818,"temperature":0.7,"pith_summary":"The paper tries to establish that a small code model reasons better at math when it is fine-tuned on Program-of-Thought solutions written in several programming languages instead of one, and better still when it can mix answers across those languages after generating them. Program-of-Thought means the model writes a short program as its intermediate reasoning step and runs it to get the answer. Using automatically drafted, execution-verified solutions in Python, C++, Java, and Matlab for GSM8K and MATH, the authors fine-tune 7-billion-parameter code models and report that each language's accuracy rises by about 2.5 points over single-language training. After a posterior mixing step that votes over the four languages and breaks ties with a scorer, the model reports roughly a 6% improvement over a single-language PoT baseline with data augmentation. The practical stakes are that multilingual program training is a cheap, general recipe for small code models on math reasoning, and that the choice of programming language is itself part of the reasoning strategy.","feed_headline":"Mixing four programming languages lifts math AI by 6 percent","feed_subtitle":"Fine-tuning on Python, C++, Java and Matlab solutions beats one-language data augmentation on math word problems.","key_machinery":"The load-bearing mechanism is a supervised fine-tuning setup that makes programming language an explicit conditional variable. For each GSM8K and MATH training problem, the authors prompt a large language model to write a solution function in Python, C++, Java, and Matlab, execute every function, and keep only those that return the correct answer; they then fine-tune a code model on the prompt asking for a program in a specified language. The resulting MultiLingPoT model can answer in any of the four languages, which enables the paper's hybrid strategies. Prior hybrids choose one language before generation using case-based similarity, small-model scorers, an LLM scorer, or direct preference optimization, while posterior hybrids generate all four programs and then mix them; the posterior voting and scorer variants carry the reported gains.","core_discovery":"On the paper's own terms, the central discovery is that supervising a model on the same math problem in four programming languages produces cross-language transfer: after fine-tuning, each individual language performs better than a model trained on that language alone, by about 2.5% on average. On complex problems, language preferences become clear, with Python best for number theory, Java for geometry, and Matlab for counting and probability, and no single language is always best. Because the MultiLingPoT model can generate all four solutions, the paper shows that posterior hybrid strategies, meaning self-consistency voting over the four generated programs and optionally an LLM-based scorer to break ties, outperform the best single-language data-augmented baseline by about 6% overall. The paper also reports that the gains transfer to other code-capable base models, while prior strategies that pick a language before generating, including direct preference optimization, are weak and sometimes near random.","pith_inferences":["If the active ingredient is solution diversity rather than multilingualism per se, a Python-only training set with many distinct correct programs per problem should reproduce most of the 6% gain; that is a direct test the paper does not run.","Because complex problems show stronger language preferences, per-category routing, such as always using Matlab for counting problems, could approach the posterior gain at a fraction of the inference cost.","The reported upper bounds in the paper suggest that most of the remaining gap comes from selecting the best language or answer, not from generating correct code, so better mixing rules are the next lever.","A cheap approximation to posterior mixing would be a lightweight scorer that reads the first one or two generated programs and decides whether the remaining languages are worth generating."],"forward_implications":["Multilingual PoT training acts as data augmentation: each language gets about a 2.5% accuracy lift, so gains come without writing new math problems.","No single programming language dominates, so language choice should be treated as part of the inference strategy rather than fixed in advance.","Posterior mixing costs four generations per question but reliably beats single-language self-consistency, improving accuracy by about 6% over single-language PoT with data augmentation.","The recipe transfers across code-capable base models, and stronger code pretraining produces larger gains.","Choosing the language before generating is hard: prior strategies are near random on simple questions, while posterior selection benefits from seeing the actual generated code."],"supporting_citations":[{"why":"Introduces Program-of-Thought prompting, the paradigm the paper extends from single-language to multilingual solutions.","marker":"Chen et al., 2022"},{"why":"Establishes program-aided language models and the code-execution approach behind the single-language PoT baselines.","marker":"Gao et al., 2023"},{"why":"Supplies GSM8K, the source of simple training and evaluation problems.","marker":"Cobbe et al., 2021"},{"why":"Supplies MATH, the source of complex training and evaluation problems.","marker":"Hendrycks et al., 2021"},{"why":"Prior multilingual PoT work whose prompt-based method motivates the paper's supervised fine-tuning approach and language-selection contribution.","marker":"Luo et al., 2024"},{"why":"Self-consistency voting, which is the core of the posterior hybrid strategy.","marker":"Wang et al., 2022"},{"why":"Provides the base code model used in the main fine-tuning experiments.","marker":"Roziere et al., 2023"},{"why":"Supplies direct preference optimization, one of the prior hybrid selection implementations evaluated.","marker":"Rafailov et al., 2024"}],"fun_headline_variants":["Multilingual code boosts math reasoning by 6%","Write math in 4 languages: AI improves by 6%","Cross-language transfer lifts math AI by 6%","Python, C++, Java, Matlab: math AI gains 6%","Multi-program reasoning outdoes single-language by 6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single-language data-augmented baseline is a fair control for data volume; because that baseline repeats identical Python solutions, the reported gains could be caused by having more varied solutions rather than by mixing programming languages.","fun_headline_variants_meta":{"raw":{"variants":["Multilingual code boosts math reasoning by 6%","Write math in 4 languages: AI improves by 6%","Cross-language transfer lifts math AI by 6%","Python, C++, Java, Matlab: math AI gains 6%","Multi-program reasoning outdoes single-language by 6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1522,"prompt_tokens":931,"completion_tokens":591,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":507}},"tokens_in":547,"tokens_out":591,"duration_ms":5506,"temperature":1.0,"reasoning_tokens":507,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:54:14.842268+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a diverse single-language control: for each GSM8K and MATH training problem, collect several distinct correct Python solutions with different algorithms, code styles, or paraphrases so the Python-only training set matches MultiLingPoT's sample count and diversity. If that model matches or beats the 6% advantage, then multilingualism itself is not the causal driver.","supporting_citations":[],"review_version":1}