Pith. sign in

REVIEW 4 major objections 3 minor 142 references

This paper claims that a strong LLM can improve by distilling from the logit difference between two weaker models, even when both are less capable than the student.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:20 UTC pith:THL3IADC

load-bearing objection A clean, likeable distillation paper whose headline weak-to-strong claim is undercut by missing OPD baselines in the very settings where all sources are genuinely weaker. the 4 major comments →

arxiv 2607.26246 v1 pith:THL3IADC submitted 2026-07-28 cs.LG

Weak-to-Strong On-Policy Distillation

Fangxu Yu , Zinan Lin , Xiaodong Liu , Weijia Xu , Michael Xu , Tianyi Zhou , Jianfeng Gao This is my paper
classification cs.LG
keywords weak-to-strongon-policy distillationproxy teacherlogit arithmeticLLM reasoningknowledge distillationreverse KLmath reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that weak-to-strong transfer need not imitate a weak teacher. Instead, the difference between two weak models' pre-softmax scores (logits) can be treated as a capability direction, added to the student's own base logits, and then distilled on the student's own rollouts. On math and code benchmarks, this proxy-teacher distillation improves an 8B student over direct on-policy distillation and, in some settings, above the domain expert itself. The claim matters because it offers a path for frontier models to keep improving when no stronger teacher exists.

Core claim

The central claim is that the logit difference z+(s) − z−(s) between a positive and a negative weak model cancels what the two share and preserves the direction along which the positive improves over the negative. Adding this direction with a coefficient α to the student's base logits yields a proxy teacher πT,α = softmax(zbase + α(z+ − z−)) that is simultaneously more capable in the target skill and distributionally close to the student. The student minimizes per-token reverse KL toward this proxy on its own rollouts. The paper argues this isolates and transfers capability rather than imitating a weak supervisor, and reports gains across three contrast types: pre-RL vs post-RL, larger vs sm

What carries the argument

The load-bearing object is the proxy teacher, a product-of-experts composition in logit space: πT,α(a|s) ∝ πbase(a|s) · (π+(a|s) / π−(a|s))^α. The ratio π+/π− is the capability reward carried by the contrast pair; subtracting logits cancels shared weakness; α controls how far the resulting distribution may move from the student's base. Distillation minimizes reverse KL on student-generated prefixes, so the teacher corrects exactly the states the student visits. A multi-teacher extension sums several such directions in one run.

Load-bearing premise

The assumption that carries the paper is that the logit difference between two weak models is a clean, transferable capability direction; if it mostly encodes style, format, or benchmark artifacts, the proxy teacher would just be a tuned interpolation and the weak-to-strong story collapses.

What would settle it

A randomized-direction control: take the same positive and negative models but replace z+ − z− at every token with a random vector of identical norm, and distill that. If the student improves as much as with the true direction, the direction itself is not the carrier; if it does not, the identity of the contrast direction is causal.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A frontier model can keep improving without any stronger teacher, as long as a contrast pair with a separable capability gap exists.
  • The method transfers RL-acquired skills to a large student without ever running RL at the student's scale.
  • Contrast directions from different sources (RL, scale, hints) can be merged in a single distillation run by adding their logit differences.
  • Out-of-domain evaluation suggests the transferred skill does not erode general ability; in the paper's experiments it improves the student on scientific reasoning and instruction following.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same recipe could mine other contrast pairs — before/after safety tuning, generalist vs specialist, native vs translated — as long as their logit difference is directionally meaningful, making weak-to-strong a general post-training primitive.
  • Since the method requires only logits, it could be run with the student's own checkpoints from different training steps as the positive/negative pair, turning self-distillation into a weak-to-strong signal without any extra model.
  • A testable extension is per-token or per-example α: the paper uses a single scalar and shows performance peaks at moderate values, so an adaptive schedule might push gains further.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces W2S-OPD, a method to distill a strong student LLM from one or more weaker models. A proxy teacher is synthesized in logit space as softmax(z_base + α(z_+ − z_−)), where the contrast pair (m_+, m_−) is intended to isolate a capability direction; the student is trained by on-policy reverse-KL distillation against this proxy teacher. Three contrast instantiations are evaluated: a post-RL 4B expert against its pre-RL initialization, two off-the-shelf base models (4B and 0.6B), and a single 4B model conditioned on correct versus wrong hints. The student is Qwen3-8B. The authors report gains over the student base on math and code benchmarks, claim W2S-OPD outperforms OPD, and present token-level analyses of which reasoning episodes are reinforced.

Significance. If the empirical claims hold, the idea is significant: it offers a way to improve frontier models without a stronger teacher, using direction/subtraction in logit space rather than direct imitation. The formal connection in Appendix B (exponential tilting / product-of-experts view) is elegant and makes the method easy to adopt. The three contrast instantiations are practical and well motivated, and the paper includes useful analysis of learning signals and runtime overhead. However, the central attribution is currently under-supported: the OPD baseline is only reported in the pre-RL/post-RL setting, where the positive teacher is not actually weaker than the student, and the pure weak-to-strong settings lack the OPD control that would isolate the effect of the contrast subtraction. In addition, the main results rely on α tuned on the evaluation benchmarks without multiple seeds or error bars, making the claimed superiority to OPD and the 'weak-to-strong' interpretation fragile.

major comments (4)
  1. [§4.2, Tables 3 and 4] In the two genuinely weak-to-strong settings (smaller/larger and correct/wrong hints), the only baseline is the student's base model; no OPD-from-the-positive-model baseline is reported. Without that control, the observed improvement cannot be attributed to the contrast subtraction z_+ − z_−. It could equally come from plain on-policy distillation toward the weak positive model, from a distributional-smoothing effect, or from the α hyperparameter. Please add an OPD baseline with the positive model (Qwen3-4B) as teacher under the identical configuration for these settings, and also an α=0 control.
  2. [§4.3.1, Figure 4 and Table 7] The main results are reported after selecting α on the same evaluation benchmarks (Figure 4), and Table 7 fixes α=1.0 for math and 0.75 for code. Figure 4 shows that the optimal α varies across benchmarks (e.g., on LiveCodeBench α=0.75 gives 26.8 while α=1.25 gives 19.7, below the OPD reference of 22.0). This makes the headline gains optimistic and does not support the abstract's claim of consistent outperformance. Report results with a validation-selected α, or provide a robustness table across α values and multiple seeds/error bars for at least the main settings.
  3. [§3.1, Eq. (2) and Abstract] The proxy teacher is anchored at the student's own base logits, so the target is partially defined by the student. The paper's interpretation that the logit difference 'isolates a capability direction' is not independently tested; the method could reduce to a tuned interpolation between the student and a weak model. Provide a control that separates the contrast from the anchor, e.g., α=0, a negative-direction control, or a random/permuted logit difference. Also, the abstract says 'keeps improving the student even when every supervision source is weaker,' but the only setting with an OPD comparison (pre-RL/post-RL, Table 2) has a positive model that is much stronger on math (48.8 vs 17.0), so it is smaller but not weaker. The pure weak settings have no OPD baseline, leaving the central claim empirically unverified.
  4. [§4.2 and Table 2] The claim that 'W2S-OPD enables the student to surpass the domain teacher' is only demonstrated in the pre-RL/post-RL setting, where the positive model is a strong domain expert (Qwen3-4B-RL: 48.8 math vs student 17.0). That is not a weak-to-strong transfer in the sense claimed in the abstract and introduction. Please clearly separate the strong-expert setting from the genuinely weak settings, and state that the OPD comparison applies only to the former. This is not a fatal issue if the weak-setting results are supported with proper baselines, but the current presentation overstates the scope of the finding.
minor comments (3)
  1. [§3.2, third bullet] Minor grammar/wording issues: 'which token-level supervision efficiently' appears to be missing a verb; the sentence should be revised.
  2. [§3.1, after Eq. (3)] 'W2S-OPD address both challenges' should be 'addresses'.
  3. [Figure 2 caption] The caption contains an incomplete phrase ('divide and ...') and would benefit from clearer labels for the logit-offset path.

Circularity Check

0 steps flagged

No circular reduction found: the proxy-teacher construction is an empirical hypothesis tested on external benchmarks; missing OPD controls are an attribution gap, not a circular step.

full rationale

The paper's derivation chain is not circular. Equation (2) defines the proxy teacher as an exponential tilt of the student's anchor, and Equations (8)-(10) are algebraic identities re-expressing that definition as KL-regularized reward maximization; they do not by themselves entail the reported accuracy gains. The headline improvements come from external evaluations (AIME/HMMT, HumanEval+/MBPP+/LCB) against the student's own base and against OPD. alpha is a single hyperparameter swept in Figure 4, not a parameter fitted to the benchmark outcome, and no evaluation label enters the proxy-teacher construction. The phrase 'isolates the capability direction' is an interpretive label for z+ - z-; whether that direction transfers to a stronger student is an empirical hypothesis that could have failed and is tested, not assumed. The only self-citations (ThinkARM in Section 4.4 and Yu et al. 2024 in background) are not load-bearing for the central improvement claim. The reviewer's concern that Tables 3-4 omit the OPD-from-positive baseline is a valid attribution/experimental-control limitation, but an omitted control is not a reduction of a prediction to its inputs; therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The only fitted free parameter is α. The 'proxy teacher' is an algorithmic construction rather than a new physical or conceptual entity with independent falsifiable handles. The load-bearing assumptions are the isolation/transferability of logit differences and the trust-region effect of the base anchor, both inherited from prior decoding-time work.

free parameters (1)
  • α (amplification coefficient) = 1.0 (math), 0.75 (code)
    Controls the strength of the injected logit difference. Tuned on the evaluation benchmarks via Figure 4 and reported in Table 7; the paper acknowledges a moderate α is optimal but does not predict the value.
axioms (4)
  • domain assumption Logit-space linear combination of models yields a valid proxy distribution that can be used as a distillation target.
    Invoked in Eq. 2 (§3.1); follows from DExperts/Tuning-by-Proxy prior work, adopted as a design choice.
  • domain assumption The logit difference z+ − z− between two weak models isolates a transferable capability direction independent of scale.
    Central mechanism stated in §3.1: 'their difference still encodes a transferable capability direction'. This is assumed, not proven.
  • domain assumption Anchoring at zbase keeps the proxy teacher distributionally adjacent to the student, stabilizing optimization.
    Assumed throughout §3.1 and formalized in Appendix B through the KL trust-region view.
  • standard math Top-K truncation with K=32 preserves almost all teacher signal.
    Appendix A justifies truncation because teacher distributions are peaked; this is a standard estimation approximation.

pith-pipeline@v1.3.0-alltime-deepseek · 24504 in / 7234 out tokens · 69925 ms · 2026-08-01T00:20:54.460422+00:00 · methodology

0 comments
read the original abstract

On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs. Prevailing approaches assume a teacher at least as capable as the student: they either distill a larger model into a smaller one, which fails at the frontier where no larger teacher exists, or consolidate multiple domain experts trained from a shared base, which requires costly training at the student's scale. We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a simple yet effective OPD framework that improves the strong student by distilling from multiple weak models. W2S-OPD constructs a proxy teacher in logit space from a contrast pair of a positive and a negative model, both smaller than the student and cheap to obtain. Their logit difference isolates the capability direction, which is added to the student's own base model, yielding a proxy teacher that couples this direction while staying distributionally adjacent to the student. The student then distills it by minimizing the per-token reverse KL on its own rollouts. We instantiate the contrast pair as i) a post-RL expert against its pre-RL initialization, isolating the skill RL instills, ii) a larger against a smaller base model, isolating the capability from scale, and iii) a small base model with correct versus wrong hints, isolating the instance-level direction toward the solution. Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker. Analysis shows different contrasts yield distinct signals: the post-RL and hint contrasts emphasize reasoning frameworks, while the scale contrast emphasizes the solving procedure. Our code will be available at https://github.com/Yu-Fangxu/W2S-OPD.

Figures

Figures reproduced from arXiv: 2607.26246 by Fangxu Yu, Jianfeng Gao, Michael Xu, Tianyi Zhou, Weijia Xu, XiaoDong Liu, Zinan Lin.

Figure 1
Figure 1. Figure 1: W2S-OPD improves Qwen3-8B using 4B models as teachers. Accuracy is averaged over 4 math reasoning and 3 code generation benchmarks. (a) With a post-RL Qwen3-4B expert, W2S-OPD beats OPD. (b) From 2 off-the-shelf base models (Qwen3-4B and 0.6B), both weaker than the student and used without training, W2S-OPD still lifts the student above itself and both sources. Reinforcement learning with verifiable reward… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of W2S-OPD. W2S-OPD synthesizes a proxy teacher and distills it into a student. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance on the math and code benchmarks over training steps. W2S-OPD improves [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance on the math and code benchmarks with different [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: A case study that the three contrasts strengthen different reasoning tokens. The shade of [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Math case study under Pre-RL/ Post-RL setting, with the errors of direct OPD marked in [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Code case study under the pre-RL/post-RL setting with the errors of direct OPD marked in [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Math case study from under the smaller/larger setting, with the errors of the Qwen3-8B [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Code case study under the smaller/larger setting, with the errors of the Qwen3-8B student [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

142 extracted references · 4 canonical work pages · 2 internal anchors

  1. [1]

    2023 , url =

    Benjamin Elizalde and Soham Deshmukh and Mahmoud Al Ismail and Huaming Wang , title =. 2023 , url =. doi:10.1109/ICASSP49357.2023.10095889 , timestamp =

    arXiv 2023

  2. [2]

    Advances in Neural Information Processing Systems , volume=

    Checklists are better than reward models for aligning language models , author=. Advances in Neural Information Processing Systems , volume=

  3. [3]

    arXiv preprint arXiv:2605.12474 , year=

    Reward hacking in rubric-based reinforcement learning , author=. arXiv preprint arXiv:2605.12474 , year=

    Pith/arXiv arXiv

  4. [4]

    2025 , note =

    Gemini 3.1 Pro , howpublished =. 2025 , note =

    2025

  5. [5]

    arXiv preprint arXiv:2602.05125 , year=

    Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks , author=. arXiv preprint arXiv:2602.05125 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  7. [7]

    arXiv preprint arXiv:2602.21628 , year=

    Rucl: Stratified rubric-based curriculum learning for multimodal large language model reasoning , author=. arXiv preprint arXiv:2602.21628 , year=

    arXiv

  8. [8]

    arXiv preprint arXiv:2602.00846 , year=

    Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis , author=. arXiv preprint arXiv:2602.00846 , year=

    Pith/arXiv arXiv

  9. [9]

    arXiv preprint arXiv:2603.16600 , year=

    Rationale matters: Learning transferable rubrics via proxy-guided critique for VLM reward models , author=. arXiv preprint arXiv:2603.16600 , year=

    arXiv

  10. [10]

    arXiv preprint arXiv:2604.13029 , year=

    Visual preference optimization with rubric rewards , author=. arXiv preprint arXiv:2604.13029 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2510.14738 , year=

    AutoRubric-R1V: Rubric-Based Generative Rewards for Faithful Multimodal Reasoning , author=. arXiv preprint arXiv:2510.14738 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2602.04649 , year=

    Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models , author=. arXiv preprint arXiv:2602.04649 , year=

    arXiv

  13. [13]

    arXiv preprint arXiv:2602.01511 , year=

    Alternating reinforcement learning for rubric-based reward modeling in non-verifiable llm post-training , author=. arXiv preprint arXiv:2602.01511 , year=

    arXiv

  14. [14]

    arXiv preprint arXiv:2510.07284 , year=

    Online rubrics elicitation from pairwise comparisons , author=. arXiv preprint arXiv:2510.07284 , year=

    arXiv

  15. [15]

    arXiv preprint arXiv:2602.10885 , year=

    Reinforcing chain-of-thought reasoning with self-evolving rubrics , author=. arXiv preprint arXiv:2602.10885 , year=

    arXiv

  16. [16]

    arXiv preprint arXiv:2511.19399 , year=

    Dr tulu: Reinforcement learning with evolving rubrics for deep research , author=. arXiv preprint arXiv:2511.19399 , year=

    Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:2508.16949 , year=

    Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general llm reasoning , author=. arXiv preprint arXiv:2508.16949 , year=

    arXiv

  18. [18]

    arXiv e-prints , pages=

    Auto-rubric: Learning to extract generalizable criteria for reward modeling , author=. arXiv e-prints , pages=

  19. [19]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    CARMO: Dynamic Criteria Generation for Context Aware Reward Modelling , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  20. [20]

    arXiv preprint arXiv:2510.07743 , year=

    Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment , author=. arXiv preprint arXiv:2510.07743 , year=

    arXiv

  21. [21]

    International Conference on Learning Representations , volume=

    Prometheus: Inducing fine-grained evaluation capability in language models , author=. International Conference on Learning Representations , volume=

  22. [22]

    Advances in Neural Information Processing Systems , volume=

    Rule based rewards for language model safety , author=. Advances in Neural Information Processing Systems , volume=

  23. [23]

    arXiv preprint arXiv:2512.20061 , year=

    Scaling Reinforcement Learning for Content Moderation with Large Language Models , author=. arXiv preprint arXiv:2512.20061 , year=

    arXiv

  24. [24]

    arXiv preprint arXiv:2602.02994 , year=

    Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation , author=. arXiv preprint arXiv:2602.02994 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:1810.08575 , year=

    Supervising strong learners by amplifying weak experts , author=. arXiv preprint arXiv:1810.08575 , year=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  27. [27]

    International Conference on Learning Representations , volume=

    On-policy distillation of language models: Learning from self-generated mistakes , author=. International Conference on Learning Representations , volume=

  28. [28]

    arXiv preprint arXiv:2312.09390 , year=

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision , author=. arXiv preprint arXiv:2312.09390 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2606.00564 , year=

    Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding , author=. arXiv preprint arXiv:2606.00564 , year=

    Pith/arXiv arXiv

  30. [30]

    arXiv preprint arXiv:2605.21924 , year=

    Visual-Advantage On-Policy Distillation for Vision-Language Models , author=. arXiv preprint arXiv:2605.21924 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2603.07079 , year=

    Entropy-aware on-policy distillation of language models , author=. arXiv preprint arXiv:2603.07079 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv preprint arXiv:2603.11137 , year=

    Scaling reasoning efficiently via relaxed on-policy distillation , author=. arXiv preprint arXiv:2603.11137 , year=

    arXiv

  33. [33]

    Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    Stable on-policy distillation through adaptive target reformulation , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    2026

  34. [34]

    arXiv preprint arXiv:2606.01249 , year=

    Trust Region On-Policy Distillation , author=. arXiv preprint arXiv:2606.01249 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2602.12275 , year=

    On-policy context distillation for language models , author=. arXiv preprint arXiv:2602.12275 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2604.13016 , year=

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe , author=. arXiv preprint arXiv:2604.13016 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2601.19897 , year=

    Self-Distillation Enables Continual Learning , author=. arXiv preprint arXiv:2601.19897 , year=

    Pith/arXiv arXiv

  38. [38]

    arXiv preprint arXiv:2605.27115 , year=

    Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation , author=. arXiv preprint arXiv:2605.27115 , year=

    Pith/arXiv arXiv

  39. [39]

    arXiv preprint arXiv:2601.20802 , year=

    Reinforcement Learning via Self-Distillation , author=. arXiv preprint arXiv:2601.20802 , year=

    Pith/arXiv arXiv

  40. [40]

    arXiv preprint arXiv:2601.18734 , year=

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models , author=. arXiv preprint arXiv:2601.18734 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2604.20244 , year=

    Hybrid policy distillation for llms , author=. arXiv preprint arXiv:2604.20244 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2402.08679 , year=

    Cold-attack: Jailbreaking llms with stealthiness and controllability , author=. arXiv preprint arXiv:2402.08679 , year=

    Pith/arXiv arXiv

  43. [43]

    arXiv preprint arXiv:2607.05394 , year=

    Weak-to-Strong Generalization via Direct On-Policy Distillation , author=. arXiv preprint arXiv:2607.05394 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:2606.11709 , year=

    RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation , author=. arXiv preprint arXiv:2606.11709 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2604.03128 , year=

    Self-distilled rlvr , author=. arXiv preprint arXiv:2604.03128 , year=

    Pith/arXiv arXiv

  46. [46]

    2014 , publisher=

    Mathematical problem solving , author=. 2014 , publisher=

    2014

  47. [47]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Schoenfeld’s anatomy of mathematical reasoning by language models , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  48. [48]

    Advances in Neural Information Processing Systems , volume=

    Reinforcement learning teachers of test time scaling , author=. Advances in Neural Information Processing Systems , volume=

  49. [49]

    Advances in Neural Information Processing Systems , volume=

    Can llms learn by teaching for better reasoning? a preliminary study , author=. Advances in Neural Information Processing Systems , volume=

  50. [50]

    Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=

    Gedi: Generative discriminator guided sequence generation , author=. Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=

    2021

  51. [51]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    FUDGE: Controlled text generation with future discriminators , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2021

  52. [52]

    Advances in Neural Information Processing Systems , volume=

    Cold decoding: Energy-based constrained text generation with langevin dynamics , author=. Advances in Neural Information Processing Systems , volume=

  53. [53]

    arXiv preprint arXiv:2006.03535 , year=

    Cocon: A self-supervised approach for controlled text generation , author=. arXiv preprint arXiv:2006.03535 , year=

    Pith/arXiv arXiv 2006

  54. [54]

    arXiv preprint arXiv:1909.05858 , year=

    Ctrl: A conditional transformer language model for controllable generation , author=. arXiv preprint arXiv:1909.05858 , year=

    Pith/arXiv arXiv 1909

  55. [55]

    Proceedings of the 28th International Conference on Computational Linguistics , pages=

    Exploring controllable text generation techniques , author=. Proceedings of the 28th International Conference on Computational Linguistics , pages=

  56. [56]

    Advances in Neural Information Processing Systems , volume=

    Generalizing verifiable instruction following , author=. Advances in Neural Information Processing Systems , volume=

  57. [57]

    arXiv preprint arXiv:2606.30406 , year=

    MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training , author=. arXiv preprint arXiv:2606.30406 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2602.15763 , year=

    Glm-5: from vibe coding to agentic engineering , author=. arXiv preprint arXiv:2602.15763 , year=

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2606.19348 , year=

    Deepseek-v4: Towards highly efficient million-token context intelligence , author=. arXiv preprint arXiv:2606.19348 , year=

    arXiv

  60. [60]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    Weak-to-strong reasoning , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  61. [61]

    arXiv preprint arXiv:2311.12022 , year=

    Gpqa: A graduate-level google-proof q&a benchmark , author=. arXiv preprint arXiv:2311.12022 , year=

    Pith/arXiv arXiv

  62. [62]

    DExperts: Decoding-time controlled text generation with experts and anti-experts , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=

  63. [63]

    arXiv preprint arXiv:2401.08565 , year=

    Tuning language models by proxy , author=. arXiv preprint arXiv:2401.08565 , year=

    Pith/arXiv arXiv

  64. [64]

    Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Contrastive decoding: Open-ended text generation as optimization , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  65. [65]

    arXiv preprint arXiv:2604.00626 , year=

    A survey of on-policy distillation for large language models , author=. arXiv preprint arXiv:2604.00626 , year=

    Pith/arXiv arXiv

  66. [66]

    arXiv preprint arXiv:2401.17256 , year=

    Weak-to-strong jailbreaking on large language models , author=. arXiv preprint arXiv:2401.17256 , year=

    Pith/arXiv arXiv

  67. [67]

    arXiv preprint arXiv:2601.02780 , year=

    Mimo-v2-flash technical report , author=. arXiv preprint arXiv:2601.02780 , year=

    Pith/arXiv arXiv

  68. [68]

    Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Incentivizing strong reasoning from weak supervision , author=. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  69. [69]

    arXiv preprint arXiv:2506.03109 , year=

    On weak-to-strong generalization and f-divergence , author=. arXiv preprint arXiv:2506.03109 , year=

    Pith/arXiv arXiv

  70. [70]

    arXiv preprint arXiv:2606.06021 , year=

    OPRD: On-Policy Representation Distillation , author=. arXiv preprint arXiv:2606.06021 , year=

    Pith/arXiv arXiv

  71. [71]

    International Conference on Learning Representations , volume=

    Minillm: Knowledge distillation of large language models , author=. International Conference on Learning Representations , volume=

  72. [72]

    forward KL , author=

    Revisiting weak-to-strong generalization in theory and practice: Reverse KL vs. forward KL , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  73. [73]

    Thinking Machines Lab: Connectionism , year =

    Kevin Lu and Thinking Machines Lab , title =. Thinking Machines Lab: Connectionism , year =

  74. [74]

    arXiv preprint arXiv:2502.01456 , year=

    Process reinforcement through implicit rewards , author=. arXiv preprint arXiv:2502.01456 , year=

    Pith/arXiv arXiv

  75. [75]

    American Invitational Mathematics Examination (AIME) 2025 , author=

    2025

  76. [76]

    arXiv preprint arXiv:2504.11456 , year=

    Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning , author=. arXiv preprint arXiv:2504.11456 , year=

    Pith/arXiv arXiv

  77. [77]

    2024 , howpublished =

    2024

  78. [78]

    arXiv preprint arXiv:2402.03898 , year=

    Distillm: Towards streamlined distillation for large language models , author=. arXiv preprint arXiv:2402.03898 , year=

    Pith/arXiv arXiv

  79. [79]

    International Conference on Learning Representations , volume=

    Livecodebench: Holistic and contamination free evaluation of large language models for code , author=. International Conference on Learning Representations , volume=

  80. [80]

    URL https://matharena

    Matharena: Evaluating llms on uncontaminated math competitions, February 2025 , author=. URL https://matharena. ai , volume=

    2025

Showing first 80 references.