Pith. sign in

REVIEW 3 major objections 4 minor 15 references

On-Policy Delta Distillation for Multilingual Math Reasoning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Delta distillation consistently beats plain on-policy distillation for math reasoning in English, Korean, and Japanese—and English-only training transfers accuracy but not the target language.

desk verdict A solid empirical extension of OPD^2 to Korean/Japanese math reasoning, with the language-shift dissociation as the genuinely new finding; needs error bars and specification details before the quantitative claims are fully load-bearing. read the letter →

arxiv 2608.05802 v1 pith:ZVDHXWBK submitted 2026-08-06 cs.CL cs.LG

Byeongho Heo , Jaehui Hwang , Sangdoo Yun , Dongyoon Han This is my paper · ORCID
classification cs.CLcs.LG
keywords on-policydistillationdeltamultilingualmathematicalreasoningcross-lingualtransfertarget-languagepreservationpost-trainingEnglish-Koreangap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether on-policy distillation, a token-level alternative to reinforcement learning for post-training large language models, still works when the task is mathematical reasoning in English, Korean, and Japanese. It claims that a variant called On-Policy $\Delta$ Distillation ($OPD^{2}$), which uses the probability gap between a post-trained teacher and its own base model as the training signal, consistently beats the original on-policy distillation in all three languages. The paper also claims that multilingual $OPD^{2}$ generally narrows the English–Korean accuracy gap. A third claim is that English-only $OPD^{2}$ can raise Korean and Japanese benchmark scores to nearly the level of multilingual training, but the model then often answers in English instead of the question language. That last finding matters because it separates transfer of reasoning ability from preservation of the target language in evaluation.

What carries the argument

The central object is the delta signal: for each sampled token, $OPD^{2}$ replaces the student log-probability in the ordinary OPD reward with the teacher's base-model log-probability, giving reward $R_t = \log \pi^*(y_t \mid x, y_{<t}) - \log \pi^*_{\mathrm{base}}(y_t \mid x, y_{<t})$. This isolates capabilities acquired during post-training, particularly reasoning, from stylistic tendencies already present in the base model. Two auxiliary components, reward centering to obtain an advantage and a conditioning criterion based on the original OPD signal, are used in all experiments to fix the delta signal's convergence point; the paper cites its companion work for their formulas. The mechanism's work is to convert teacher rollout feedback into token-level rewards that can be applied through the on-policy update in the paper's Eq. 3.

What would settle it

Run the released $OPD^{2}$ code on Qwen3-1.7B with the same multilingual dataset and measure PolyMath Japanese accuracy and the Japanese response rate in non-thinking mode; the paper predicts 52.0 average accuracy and 90.9 percent Japanese responses. If the code reproduces OPD's numbers rather than $OPD^{2}$'s, or if the response language detector labels English-heavy outputs as Japanese, the claimed advantage and the language-shift finding would not hold.

Watch

Extended reading notes

Core claim

Using Qwen3-1.7B and Qwen3-8B as students with Qwen3-30B-A3B-2507 as teacher, and training on 100K questions in a 1:1:1 English/Korean/Japanese mix, the paper finds that $OPD^{2}$ outperforms OPD in every language and setting tested. For Qwen3-1.7B non-thinking average on PolyMath plus Global-MGSM, $OPD^{2}$ reaches 63.6 in English, 51.9 in Korean, and 52.0 in Japanese, versus 61.5, 48.8, and 48.0 for OPD. In thinking mode, $OPD^{2}$ cuts the English–Korean gap on six of seven benchmarks while leaving the seventh unchanged. The paper's additional claim is that English-only $OPD^{2}$ transfers accuracy to Korean and Japanese, averaging 52.6 and 53.4 non-thinking, close to multilingual $OPD^{2}$'s 51.9 and 52.0, but the target-language response rate drops sharply—for example, Korean from 90.5 percent to 48.3 percent and Japanese from 90.9 percent to 29.6 percent in non-thinking mode. Therefore benchmark accuracy alone overstates cross-lingual capability when the response language is not checked.

Load-bearing premise

The whole comparison rests on the assumption that the linked implementation actually performs the $OPD^{2}$ update described in the companion paper, and that the automatic language check correctly identifies whether answers are in Korean or Japanese.

Editorial extensions

If this is right

  • OPD^2 can be used as a direct replacement for OPD in multilingual reasoning post-training, giving consistent gains across model sizes and languages.
  • Multilingual OPD^2 narrows the English–Korean gap on most benchmarks without sacrificing English accuracy, so cross-lingual disparity is partly a post-training problem.
  • English-only delta distillation transfers reasoning accuracy to Korean and Japanese, which means reasoning skills learned in English are not language-bound.
  • A model can score near-multilingual accuracy while answering in English, so evaluations that only report accuracy will miss this failure mode.
  • Multilingual training data remains necessary when the goal is to answer in the user's language, even if English-only training matches accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to apply English-only OPD^2 to other language pairs and measure both accuracy and target-language rate; the paper's numbers predict large accuracy transfer but small language-fidelity transfer.
  • Because the delta signal uses the base model as a stylistic baseline, the English-only transfer should depend on how much reasoning versus style the base model already encodes in the target language.
  • Since the paper omits the centering and conditioning formulas, a reimplementation from the text alone may not reproduce the gains; the linked code, not the paper, is the actual specification of OPD^2.
  • For deployment, adding a response-language filter or an explicit language instruction could close the gap exposed by the language-shift result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper experimentally compares On-Policy Distillation (OPD) and its delta variant OPD^2 for multilingual mathematical reasoning. Using Qwen3-1.7B and Qwen3-8B as students, a Qwen3-30B-A3B-2507 teacher, 100K multilingual (English/Korean/Japanese 1:1:1) and English-only training sets, and evaluations on PolyMath, Global-MGSM, HRM8K, and MAWPS, the authors report that OPD^2 outperforms OPD on average in all three languages and in both thinking and non-thinking modes, narrows the English–Korean gap on most benchmarks, and that English-only OPD^2 transfers accuracy to Korean and Japanese while reducing target-language response rates. The paper positions OPD^2 as a drop-in post-training upgrade and argues that benchmark gains from English-only training do not imply that the model will respond in the user's language.

Significance. If the empirical claims hold, the paper is a useful extension of OPD research to East Asian languages and a clear demonstration that cross-lingual accuracy transfer and target-language preservation are separable. The explicit research questions, the multilingual data construction, the English–Korean gap analysis, and the response-language metric are valuable, and the linked official OPD^2 implementation is a reproducibility asset. The reported tables are internally consistent with the central claims. However, the absence of variance reporting or significance testing, together with incomplete specification of the OPD^2 objective and of the language-detection procedure, means the paper currently supports directional findings rather than a strong quantitative conclusion.

major comments (3)
  1. [Section 3.2, Figure 1, Tables 2–10] The claim that OPD^2 'consistently outperforms' OPD is based on single runs with no seeds, confidence intervals, or significance tests reported. Several advantages are small, such as the Qwen3-8B English non-thinking PolyMath+Global-MGSM average of 70.5 vs. 69.8 in Figure 1(b), and Table 8 shows a per-benchmark exception (GMGSM-EN: 98.4 for OPD vs. 98.0 for OPD^2). Without repeated runs or a statistical procedure, the central quantitative claim is not established. Please add variance information or explicitly reframe the result as point estimates.
  2. [Section 2.1, Eq. (4)] The OPD^2 reward in Eq. (4) is not the complete training objective: reward centering and the conditioning criterion are mentioned but deferred to the authors' prior paper [4] ('we omit their detailed formulations due to space constraints'). Since the paper's main comparison is OPD vs. OPD^2, the manuscript should include the full objective or an appendix with the exact formulas for both components. Without this, the comparison is not self-contained and any deviation between the intended method and the released implementation cannot be detected by a reader.
  3. [Section 3.4, Table 1] The target-language response rate is load-bearing for the language-shift claim, but its measurement is underspecified. The paper does not state what language-identification method or threshold is used, how code-switched responses are classified, or how the after-</think> segmentation is performed. Because Table 1 reports large drops (e.g., non-thinking Japanese from 90.9% to 29.6%), the detector's accuracy is material to the conclusion. Please specify the metric, the detector, and its treatment of mixed-language outputs.
minor comments (4)
  1. [Appendix A, Tables 2, 4, 5, 6, 8, 9, 10] Several tables contain malformed numeric entries with missing spaces or collapsed columns, e.g., Table 5 base row 'Qwen3-1.7B36.3 93.7 90.592.652.8 55.371.670.4' and Table 8 OPD row '41.298.493.3 95.2 63.9 67.258.974.0'. Please repair the formatting so that every value is readable.
  2. [Section 3.1] The paper does not state whether any decontamination or overlap check was performed between the Nemotron training samples and the PolyMath, Global-MGSM, HRM8K, and MAWPS evaluation sets. Even though the OPD vs. OPD^2 comparison would be less affected, the absolute transfer numbers in Section 3.4 could be inflated by overlap; please report the overlap analysis or discuss the risk.
  3. [Section 3.1] Decoding and evaluation details are missing: the paper does not specify greedy vs. sampled decoding, temperature, number of samples, or the answer-extraction procedure for thinking-mode outputs. Please provide these details for reproducibility.
  4. [Section 2, Eq. (1)] The notation D_KL(πθ(y|x)‖π*(y|x)) may confuse readers because Eq. (2) defines the reward as log π* − log πθ, which is the negative of the KL gradient. A one-sentence explanation connecting Eq. (1), Eq. (2), and Eq. (3) would improve readability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central OPD2-versus-OPD claims are empirical measurements on new multilingual benchmarks; the only self-citation is a normal method-dependency for omitted implementation details.

full rationale

The paper's derivation chain is minimal: it restates the standard OPD objective (Eq. 1) and token-level reward (Eq. 2), then defines the OPD2 delta reward in Eq. 4 as log pi*(y_t) - log pi*_base(y_t). None of these equations defines an output in terms of the benchmark numbers being predicted; Eq. 4 is an externally defined learning signal, not a quantity fitted to the reported accuracies. The load-bearing claims—"OPD2 consistently outperforms the original OPD" and "English-only OPD2 ... often shifts the responses toward English"—are empirical comparisons measured on PolyMath, Global-MGSM, HRM8K, MAWPS, and the response-language rates in Table 1. No fitted parameter is renamed as a prediction. The only self-citation is in Section 2.1, where the paper says it incorporates reward centering and a conditioning criterion but omits their formulations "due to space constraints" and refers the reader to [4], the authors' own prior work. This is a reproducibility and verification dependency rather than a circular reduction: the current paper links the official OPD2 implementation and the empirical OPD2-vs-OPD comparison is not derived from [4] but from new experiments. The unspecified language detector is likewise an external verifiability concern, not a circular step. There is no uniqueness theorem imported, no ansatz smuggled in via citation, and no renaming of a known result presented as derivation. The central contribution is self-contained empirical evidence, so the circularity score is low.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted constants enter the central comparison; the main assumptions are about the validity of the OPD^2 signal and the reliability of the data and language detector.

assumptions (3)
  • domain assumption The teacher base model's log-probabilities isolate post-training reasoning capabilities, so the delta signal measures only what post-training added.
    Section 2.1 defines the OPD^2 reward as log pi* minus log pi*_base. If the base model already contains multilingual style or response-language biases, the delta signal does not cleanly separate reasoning transfer from language preference.
  • domain assumption The training data (Nemotron) does not overlap with evaluation benchmarks (PolyMath, Global-MGSM, HRM8K, MAWPS).
    Leakage would inflate the reported accuracy gains; the paper states no overlap across training languages but does not test against evaluation sets.
  • domain assumption The two corrective components of OPD^2 (reward centering and conditioning criterion) from ref [4] are correctly instantiated in the experiments.
    Section 2.1 omits their formulas and defers to [4], so the empirical comparison assumes the reference implementation is the method being claimed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On-Policy Delta Distillation for Multilingual Math Reasoning." pith.science (2026). https://pith.science/paper/ZVDHXWBK

@misc{pith2026260805802,
  author       = {Pith},
  title        = {Pith review of: On-Policy Delta Distillation for Multilingual Math Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZVDHXWBK}},
  note         = {Machine review of arXiv:2608.05802}
}
abstract

On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, On-Policy Delta Distillation (OPD$^2$), for mathematical reasoning in English, Korean, and Japanese. OPD$^2$ improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD$^2$ consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese, and generally narrows the English-Korean performance gap. We further find that English-only OPD can also increase performance for Korean and Japanese, but often shifts the responses toward English, highlighting the importance of multilingual data to preserving target-language responses.

Figures

Figures reproduced from arXiv: 2608.05802 by the authors.

Figure 1
Figure 1. Performance comparison for OPD and OPD2 . The graph shows the mathematical reasoning benchmark results for PolyMath and Global-MGSM in three languages. OPD and OPD2 substantially improve the performance for all languages. 3 Experiments We evaluate the improvements in mathematical reasoning achieved through OPD and OPD2 training. We use instruction-tuned models and investigate whether OPD remains effective in multili… view at source ↗
Figure 2
Figure 2. English–Korean accuracy gap. The graph shows the math performance difference for English and Korean in thinking mode. Multilingual OPD generally narrows the gap, although the effect varies across benchmarks. Base EN-only OPD² Multilingual OPD² English Korean Japanese (a) Thinking mode English Korean Japanese (b) Non-thinking mode P oly M a t h + Glo b al-M G S M Av g 70 65 60 55 50 65.0 63.8 66.4 55.1 57.6 59.2 55.7… view at source ↗
Figure 3
Figure 3. English-only OPD2 versus multilingual OPD2 . We compare OPD2 performance for the English-only dataset and the multilingual dataset. Surprisingly, English-only OPD2 also improves the KO and JA performance, which is comparable to multilingual OPD2 in non-thinking mode. Qwen3-8B. The advantage of the delta signal is therefore at least as pronounced in the non-English languages as in English. This suggests that OPD2 doe… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages

  1. [4]

    On-policy delta distillation.arXiv preprint arXiv:2607.15161, 2026

    Byeongho Heo, Jaehui Hwang, Sangdoo Yun, and Dongyoon Han. On-policy delta distillation.arXiv preprint arXiv:2607.15161, 2026

    arXiv 2026

  2. [1]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    arXiv 2025

  3. [2]

    Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780, 2026

    Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780, 2026

    arXiv 2026

  4. [3]

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026

    arXiv 2026

  5. [5]

    Entropy-aware on-policy distillation of language models

    Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. InF orty-third International Conference on Machine Learning, 2026

    work page 2026

  6. [6]

    A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626, 2026

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626, 2026

    arXiv 2026

  7. [7]

    Crosslingual on-policy self- distillation for multilingual reasoning.arXiv preprint arXiv:2605.09548, 2026

    Yihong Liu, Raoyuan Zhao, Michael A Hedderich, and Hinrich Schütze. Crosslingual on-policy self- distillation for multilingual reasoning.arXiv preprint arXiv:2605.09548, 2026

    arXiv 2026

  8. [8]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    arXiv 2024

Show all 15 references
  1. [9]

    Nemotron-sft-multilingual-v2

    NVIDIA. Nemotron-sft-multilingual-v2. https://huggingface.co/datasets/nvidia/Nemotron-SFT -Multilingual-v2, 2026. Hugging Face dataset

    2026

  2. [10]

    Nemotron-math: Efficient long-context distillation of mathematical reasoning from multi-mode supervision.arXiv preprint arXiv:2512.15489, 2025

    Wei Du, Shubham Toshniwal, Branislav Kisacanin, Sadegh Mahdavi, Ivan Moshkov, George Armstrong, Stephen Ge, Edgar Minasyan, Feng Chen, and Igor Gitman. Nemotron-math: Efficient long-context distillation of mathematical reasoning from multi-mode supervision.arXiv preprint arXiv...

    2025

  3. [11]

    Polymath: Evaluating mathematical reasoning in multilingual contexts.Advances in Neural Information Processing Systems, 38, 2026

    Yiming Wang, Pei Zhang, Jialong Tang, Hao-Ran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, et al. Polymath: Evaluating mathematical reasoning in multilingual contexts.Advances in Neural Information Processing Systems, 38, 2026

    2026

  4. [12]

    Language models are multilingual chain-of-thought reasoners.arXiv preprint arXiv:2210.03057, 2022

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush V osoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought reasoners.arXiv preprint arXiv:2210.03057, 2022

    arXiv

  5. [13]

    Benchmax: A comprehensive multilingual evaluation suite for large language models.arXiv preprint arXiv:2502.07346, 2025

    Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, and Fei Yuan. Benchmax: A comprehensive multilingual evaluation suite for large language models.arXiv preprint arXiv:2502.07346, 2025

    2025 arXiv

  6. [14]

    Understand, solve and translate: Bridging the multilingual mathematical reasoning gap

    Hyunwoo Ko, Guijin Son, and Dasol Choi. Understand, solve and translate: Bridging the multilingual mathematical reasoning gap. InProceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), pages 78–95, 2025

    2025

  7. [15]

    Verification of chain-of-thought prompting in japanese

    Kaito Horio, Eiki Murata, Hao Wang, Tatsuya Ide, Daisuke Kawahara, Takato Yamazaki, Kenta Shinzato, Akifumi Nakamachi, Shengzhe Li, and Toshinori Sato. Verification of chain-of-thought prompting in japanese. InProceedings of the Annual Conference of JSAI, JSAI2023: 3T1GS602–3T...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.