Pith. sign in

REVIEW 4 major objections 6 minor 67 references

This study claims RL-trained LLMs can be merged with a 7% average loss, versus 19% for SFT-trained models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:59 UTC pith:JMUFTAVU

load-bearing objection A broad and genuinely new empirical sweep on post-training paradigm and model merging, but the headline causal claim about RL is undercut by the SFT warm-start and unmatched hyperparameters; worthwhile to read, not yet convincing on the mechanism. the 4 major comments →

arxiv 2607.22039 v1 pith:JMUFTAVU submitted 2026-07-24 cs.CL

Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs

Zixuan Ren , Jinliang Lu , Junhong Wu , Yang Zhao , Dai Dai , Hua Wu , Haifeng Wang , Chengqing Zong This is my paper
classification cs.CL
keywords model mergingtask conflictsreinforcement learningsupervised fine-tuningtask vectorsparameter interferenceadvantage decayLLM post-training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the post-training paradigm of a specialist language model—supervised fine-tuning (SFT) versus reinforcement learning (RL)—determines how much its parameters conflict with another specialist's when the two are merged into one model. On five verifiable tasks and with four merging methods, RL-trained models preserve performance far better after merging: the average relative drop is about 7% for RL and 19–22% for SFT, with some SFT tasks collapsing by 65%. If true, task conflicts are not inherent to merging; they are shaped by how the models were trained, and RL becomes a practical route to building generalist LLMs by fusing specialists without retraining. The paper proposes three mechanisms—smaller gradient updates from on-policy data, an advantage term that decays as the model converges ('enough is as good as a feast'), and joint optimization over positive and negative examples—and supports them with update-norm measurements, a conflict-norm statistic, and a convergence bound on the advantage.

Core claim

The central claim is that RL-trained LLMs are substantially more suitable for merging than SFT-trained ones, largely preserving individual performance after parameter fusion. In the paper's own numbers, SFT models lose on average 19% relative performance under TIES merging while GRPO-trained models lose 7.1%; instruction-following tasks, which collapse by 25–29% under SFT, lose essentially nothing under RL. The authors attribute the advantage to three mechanisms: (1) on-policy data yield smaller gradient-update norms, so RL training is less likely to overwrite knowledge from other tasks; (2) the RL objective is adaptive—advantage scales updates and provably converges to zero as reward approa

What carries the argument

The central mechanism is advantage-scaled gradient descent combined with on-policy sampling: RL multiplies each per-sample gradient by an advantage A = r - baseline, so the effective update is η·A·∇log π; SFT applies η·∇log π with fixed intensity. The paper proves (Theorems 1–2) that E|A| ≤ sqrt(Var(r)) and that E|A|→0 as training converges, meaning RL updates shrink automatically. The accompanying conflict norm, ||C(Δθ_t1,Δθ_t2)|| = L2 norm of element-wise products of two task-update vectors restricted to negative entries, quantifies how much two tasks push parameters in opposite directions; the paper uses this machinery to claim RL reduces conflict both because |A| decays and because posit

Load-bearing premise

The load-bearing premise is that the SFT-versus-RL comparison isolates the training objective—but the RL arm is initialized by SFT warm-up on the same task data, trained with a five-times smaller learning rate, and regularized toward the reference model, so the smaller, more mergeable updates may come from those confounds rather than from RL itself.

What would settle it

Train SFT under RL's exact hyperparameters (same warm-up init, learning rate 1e-6, KL/weight-decay regularization) and merge it: if the post-merge drop falls to RL-like levels, the mechanism story is falsified though the practical headline survives. Alternatively, compare conflict norms per unit update: if RL and SFT have equal conflict at matched total update magnitude, the 'task-orthogonal subspace' claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, practitioners can merge RL-trained task specialists with far less performance loss than SFT specialists, across merging methods—even simple parameter averaging.
  • The benefit generalizes across RL algorithms (PPO, GRPO, REINFORCE++) and base model scales (3B to 24B), so the effect is attributable to RL post-training broadly, not to a specific recipe.
  • Multi-model merging scales gracefully on RL models: per-task drops stay gradual as more models are fused, whereas SFT models degrade steeply, supporting RL as a basis for generalist-model construction.
  • Because RL models show fewer conflicting parameter signs (especially among high-magnitude parameters), merging methods that keep only large task-vector entries (TIES, DARE) should become more reliable on RL-trained models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the mechanism attribution is confounded—RL runs are initialized by SFT warm-up, use a 5× smaller learning rate, and add KL regularization, and the RFT ablation already cuts update norms 2–3× without any RL objective or negative samples; so the practical headline may survive even if the three-factor explanation is wrong.
  • Editorial inference: a cheap falsifying experiment—SFT with matched learning rate, same init, and a KL/weight-decay regularizer—could tell whether 'small updates' or 'RL objective' is the true cause; if matched-budget SFT becomes mergeable, the paper's mechanism story reduces to a regularization story.
  • Editorial inference: the conflict norm measures element-wise sign disagreement; measuring cosine similarity between task vectors or linear mode connectivity along merge paths would test whether RL truly yields task-orthogonal subspaces or simply smaller updates with the same alignment pattern.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper asks whether the post-training paradigm (SFT vs RL) changes how well individually fine-tuned LLMs can be merged. Across five verifiable tasks, three base models, three RL algorithms, and four merging strategies, it reports that RL-trained specialists retain substantially more performance after merging than SFT-trained specialists (e.g., Table 1: TIEs average drop 7.1% vs 19%). It then proposes three mechanisms: (1) RL's on-policy data reduce update norms; (2) RL's advantage scaling makes updates vanish at convergence; (3) joint optimization of positive and negative samples produces less conflicting parameter updates. The mechanistic claims are supported by update-norm measurements, conflict-norm curves, a landscape visualization, and an RL-Pos ablation.

Significance. If the headline result were established under controlled conditions, this would be an important and practically useful finding: it would identify the training paradigm as a first-order determinant of model mergeability and point to RL post-training as a way to build mergeable specialist suites. The paper's empirical breadth is a genuine strength: 5 tasks, 3 base models, 3 RL algorithms, 4 merging methods, with the headline effect measured externally on merged-model performance rather than derived from the theory. The central empirical claim is therefore not circular. However, a load-bearing confound in the main comparison (RL warm-start, lower LR, KL regularization) means the paper's causal attribution to the RL objective is not currently identified. The paper is transparent enough to expose this (F.1, F.2), which is to its credit, but the missing controls are decisive for the mechanism story.

major comments (4)
  1. [§3.2 / Table 1, with F.1–F.2] The headline comparison is confounded by initialization and regularization. F.1 states that before RL the authors 'first fine-tune the base model on the math, code, and instruction-following datasets,' so for at least these three tasks each RL specialist starts from the same multi-task SFT checkpoint, whereas each SFT specialist starts from the pretrained base. The task vectors τ_i = θ_i − θ_0 in Eq. (1) therefore share the large common component θ_warm − θ_0 for all RL models; merging them is equivalent to merging small deltas around a common center, which can yield small drops regardless of the RL objective. The RL arm also uses a 5× lower learning rate (1e-6 vs 5e-6) and KL regularization (1e-3) (F.2), with no matched compute budget. This is not a local issue: it affects the central comparison in Table 1 and the 'RL objective mitigates task conflicts' conclusion. I would need a contro
  2. [§4.2, Theorems 1–2, Eq. (8), Appendix B] The 'vanishing advantage' mechanism is largely definitional. Theorem 2 assumes E(r_n) → r* (Appendix B.1, Eq. 13) and then concludes E|A_n| → 0; that is a restatement of convergence, not a demonstrated property of RL training on these models. The update-magnitude formula Δθ_RL = Σ η A_s G_s in §4.2 omits the KL-regularization gradient (coefficient 1e-3) and the PPO/GRPO clipping or normalization in Eq. (4), both of which are present in the actual training and do not vanish with the advantage. Figure 5/6 plot norms over 120 steps but there is no evidence that the RL runs have converged or that the advantage has become small relative to the KL term. To support factor (2), the authors should measure the actual per-step update norms and conflict norms from the optimizer state (including KL and clipping), and show that the advantage term itself is the dominant cause of the reduced updates, no
  3. [§3.2, Tables 1, 4, 5; Figures 2, 3] All reported numbers are single runs; there are no seeds, error bars, or significance tests. Given the empirical claim is a comparative statement about training paradigms, at least the headline comparisons (e.g., Table 1 TIEs row, Figure 3) need multiple seeds and a measure of dispersion. Some gaps are narrow (e.g., Puzzle under averaging: SFT −65% vs RL −56%), and without variance the 'substantially' claim is not fully quantified. This is fixable but should be addressed before the empirical section can be relied upon.
  4. [§4.3 / Figure 7] The RL-Pos ablation is a reasonable way to isolate negative samples, but it does not support the paper's subspace claim: no measurement of the 'unbiased task-specific parameter subspace' is provided. Figure 7 shows only merged-performance drops; H1/H2 are tested, but the proposed mechanism (negative examples steer updates to an unbiased subspace) is not directly verified. Also, since RL-Pos is trained with the same warm-start and KL regularization, the comparison with full RL is clean for the negative-sample factor but still inherits the warm-start confound relative to SFT. The section should either present direct evidence about the parameter subspace or weaken the claim.
minor comments (6)
  1. [Figure 1 caption] The caption reads 'Comparison of performance changes between SFT and RFT in model merging,' but the surrounding text refers to RL-trained models. This appears to be a typo; please correct.
  2. [Table 1] The unmerged SFT average is listed as 61.5, but the five per-task values (61.9, 60.5, 63.9, 86.2, 52.8) average to about 65.1. Please check the arithmetic or clarify how the average is computed.
  3. [Appendix F.1] The warm-start sentence names only math, code, and instruction-following datasets, while the main-text Table 1 also includes puzzle and ranking. Please clarify whether the RL models for those tasks are warm-started, and if so, on what data; if not, the design is inconsistent across the five tasks.
  4. [§3.3, Eq. (6)] The notation L(·) is used for a 'performance function' but is conventionally a loss. Please define whether higher L is better and whether it is accuracy or loss.
  5. [§4.2, Eq. (11)] The '≪' relation is used without a formal scale. Since this is a heuristic statement, consider stating it as an empirical finding or proving a bound with explicit constants.
  6. [§3.1 / Figure 1] The text says the pairwise merging setup focuses on two models at a time, but Figure 1 shows results for up to five merged models. Please clarify how the two settings relate.

Circularity Check

1 steps flagged

Factor (2) reduces to the convergence assumption; the headline measurement itself is external.

specific steps
  1. self definitional [Section 4.2, Theorem 2 / Eq. (8); Appendix B.1 Eq. (13)]
    "Based on theorem 1, the advantage estimate A_n(a,x) at the n-th step converges to zero in expectation. Since rewards are bounded within a fixed interval and advantages have zero mean for each state, we obtain lim_{n→∞} E(|A_n(a,x)|) = 0."

    Factor (2) claims that RL's 'intrinsic dynamics' progressively reduce update magnitude and conflicts as the model converges. The proof of Theorem 2 imports global convergence, lim_n E(r_n)=r* (Eq. 13, citing Zhang et al. 2020), applies Bhatia-Davis to get Var(r_n)->0, and concludes E|A_n|->0. Thus 'vanishing advantage' is a restatement of the convergence assumption, not an independent property that distinguishes RL from SFT (SFT gradients also vanish at a local optimum). This 'adaptive update' is then used in Eq. (11) to assert E||C||_RL << E||C||_SFT, so the theoretical explanation of lower RL conflict is definitionally tied to the assumed convergence rather than empirically derived.

full rationale

The headline empirical finding (Table 1) is an external measurement of merged-model performance and does not reduce to the theory; no fitted parameter is renamed as a prediction and no self-citation is load-bearing. The circular content is confined to the mechanism section: factor (2) is a convergence tautology. Separately, Appendix F.1's shared multi-task SFT warm-up for all RL specialists is a confound (all task vectors share θ_warm−θ_base), but this is a validity threat to the causal attribution rather than a definitional equivalence, so it is noted here rather than scored as an additional circular step. Overall score 4: partial circularity in one of the three explanatory factors, with the central empirical claim otherwise independent.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No free parameters are fitted to data in the central claim. Training hyperparameters (LR 1e-6/5e-6, batch 512/128, KL 1e-3) and merging hyperparameters (TIEs sensitivity 0.8, interpolation 0.5, Task-Arithmetic lambda 0.7) are standard fixed choices from prior work, not tuned to manufacture the SFT-RL gap, though they remain free choices that could affect the comparison. The conflict indicator matrix C (Eq 9) and conflict norm (Eq 10) are new measurement definitions over existing quantities, not invented entities requiring independent falsification.

axioms (5)
  • domain assumption lim E(r_n) = r* (the RL algorithm globally converges to the optimal expected reward)
    Appendix B.1, Eq (13), citing Zhang et al. (2020) for policy-gradient convergence. Not established for GRPO/PPO on LLMs with KL penalties and clipping; Theorem 2—the basis of factor (2)—depends entirely on it.
  • domain assumption Reward is bounded in a fixed interval [a, r*] (all five tasks use verifiable/binary rewards)
    Appendix B.1-B.2; required for the Bhatia-Davis variance decay, hence for lim E|A_n| = 0.
  • ad hoc to paper RL weight updates are well approximated by eta * sum A_s*G_s, with KL term and PPO/GRPO clipping omitted
    Section 4.2 norm and conflict-norm derivations. The actual GRPO update includes a KL-regularization term (coefficient 1e-3) that pulls the model toward the reference and may be the real driver of small updates; omitting it makes the formal inequality simple but the mapping to real training loose.
  • domain assumption Task vectors Delta_theta = theta_ti - theta_0 and their Hadamard products faithfully measure merge-relevant conflicts
    Sections 2.1, 4.2 Eq (9)-(10); inherited from Ilharco et al. (2023) and Yadav et al. (2023). Assumed rather than validated for these models/tasks.
  • standard math Bhatia-Davis inequality, E|X| <= sqrt(E[X^2]), and convex-analysis results (Krein-Milman, extreme points of the mean-constrained simplex)
    Appendix B.1-B.2; standard results, not in question.

pith-pipeline@v1.3.0-alltime-deepseek · 4364 in / 4458 out tokens · 249810 ms · 2026-08-01T05:59:37.679154+00:00 · methodology

0 comments
read the original abstract

Model merging plays a crucial role in consolidating multiple specialized models into a single, unified model, especially in the era of large language models (LLMs). Recent research has primarily focused on developing strategies to enhance merging performance with the trained models, while the impact of training paradigms, such as supervised fine-tuning (SFT) and reinforcement learning (RL), on the effectiveness of model merging remains underexplored. In this study, we systematically explore the merging behavior of RL-trained LLMs compared to those trained with traditional SFT. Through comprehensive evaluations across five representative tasks, we find that RL significantly reduces task conflicts and results in less performance degradation after merging, making RL-trained models particularly well-suited for this process. To unearth the reasons behind the superior suitability of RL for model merging, we conduct extensive empirical experiments and theoretical analyses. Our findings highlight three key factors: (1) On-policy training data in RL control the gradient updates in a smaller magnitude, reducing the risk of overwriting existing knowledge for other tasks in the model. (2) The RL optimization objective, which favors ``\textit{enough is as good as a feast}", progressively reduces the magnitude and the number of conflict parameter updates as the model converges. (3) Joint optimization of positive and negative examples in RL steers the model towards an unbiased task-specific parameter subspace, ensuring robust performance while further preventing parameter conflicts.

Figures

Figures reproduced from arXiv: 2607.22039 by Chengqing Zong, Dai Dai, HaiFeng Wang, Hua Wu, Jinliang Lu, Junhong Wu, Yang Zhao, Zixuan Ren.

Figure 1
Figure 1. Figure 1: Comparison of performance changes between SFT and RFT in model merging. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Performance comparison across three tasks using TIEs merging, applied to both SFT [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance comparison across three tasks using TIEs merging, applied to both SFT and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance landscape visualization around [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The norm of SFT and RL 20 40 60 80 100 120 Training steps 0 50 100 150 200 250 ||C||c o nflict (* 1 e-6) math&code - RL code&IF - RL math&IF - RL math&code - SFT code&IF - SFT math&IF - SFT [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Performance drop in cross-task merg￾ing for RL and RL-Pos models under model av￾eraging and TIEs. 5 RELATED WORKS 5.1 MODEL MERGING OF LARGE LANGUAGE MODELS Model merging offers a practical alternative to retraining by eliminating the need for access to raw training data or computationally expensive fine-tuning procedures (Ilharco et al., 2022; Crisostomi et al., 2024; Yang et al., 2024a). In this work, we… view at source ↗
Figure 8
Figure 8. Figure 8: Proportion of parameter sign conflicts under varying proportions of retained parameters. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The full content of performance landscape [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 32 linked inside Pith

  1. [1]

    Advances in Neural Information Processing Systems , volume=

    C^2M^3 : Cycle-Consistent Multi-Model Merging , author=. Advances in Neural Information Processing Systems , volume=

  2. [2]

    arXiv preprint arXiv:2408.07666 , year=

    Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities , author=. arXiv preprint arXiv:2408.07666 , year=

    Pith/arXiv arXiv

  3. [3]

    Advances in Neural Information Processing Systems , volume=

    Ties-merging: Resolving interference when merging models , author=. Advances in Neural Information Processing Systems , volume=

  4. [4]

    arXiv preprint arXiv:2312.04339 , year=

    Merging by matching models in task parameter subspaces , author=. arXiv preprint arXiv:2312.04339 , year=

    Pith/arXiv arXiv

  5. [5]

    2025 , url=

    Lu Li and Tianyu Zhang and Zhiqi Bu and Suyuchen Wang and Huan He and Jie Fu and Yonghui Wu and Jiang Bian and Yong Chen and Yoshua Bengio , booktitle=. 2025 , url=

    2025

  6. [6]

    arXiv preprint arXiv:2405.07813 , year=

    Localizing task information for improved model merging and compression , author=. arXiv preprint arXiv:2405.07813 , year=

    Pith/arXiv arXiv

  7. [7]

    arXiv preprint arXiv:2212.04089 , year=

    Editing models with task arithmetic , author=. arXiv preprint arXiv:2212.04089 , year=

    Pith/arXiv arXiv

  8. [8]

    2025 , url=

    Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts , author=. 2025 , url=

    2025

  9. [9]

    CoRR , year=

    Cross-task linearity emerges in the pretraining-finetuning paradigm , author=. CoRR , year=

  10. [10]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Linear Mode Connectivity and the Lottery Ticket Hypothesis , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , editor =

    2020

  11. [11]

    Advances in Neural Information Processing Systems , volume=

    Ensemble of averages: Improving model selection and boosting performance in domain generalization , author=. Advances in Neural Information Processing Systems , volume=

  12. [12]

    Advances in Neural Information Processing Systems , volume=

    Diverse weight averaging for out-of-distribution generalization , author=. Advances in Neural Information Processing Systems , volume=

  13. [13]

    arXiv preprint arXiv:2001.02312 , year=

    Stochastic weight averaging in parallel: Large-batch training that generalizes well , author=. arXiv preprint arXiv:2001.02312 , year=

    Pith/arXiv arXiv 2001

  14. [14]

    arXiv preprint arXiv:1803.05407 , year=

    Averaging weights leads to wider optima and better generalization , author=. arXiv preprint arXiv:1803.05407 , year=

    Pith/arXiv arXiv

  15. [15]

    2025 , eprint=

    Generalizing Verifiable Instruction Following , author=. 2025 , eprint=

    2025

  16. [16]

    Advances in neural information processing systems , volume=

    Visualizing the loss landscape of neural nets , author=. Advances in neural information processing systems , volume=

  17. [17]

    2025 , url=

    Tianzhe Chu and Yuexiang Zhai and Jihan Yang and Shengbang Tong and Saining Xie and Dale Schuurmans and Quoc V Le and Sergey Levine and Yi Ma , booktitle=. 2025 , url=

    2025

  18. [18]

    arXiv preprint arXiv:2309.14316 , year=

    Physics of language models: Part 3.1, knowledge storage and extraction , author=. arXiv preprint arXiv:2309.14316 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2407.20311 , year=

    Physics of language models: Part 2.1, grade-school math and the hidden reasoning process , author=. arXiv preprint arXiv:2407.20311 , year=

    Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:2410.01769 , year=

    Quantifying generalization complexity for large language models , author=. arXiv preprint arXiv:2410.01769 , year=

    Pith/arXiv arXiv

  21. [21]

    What Do Learning Dynamics Reveal About Generalization in

    Katie Kang and Amrith Setlur and Dibya Ghosh and Jacob Steinhardt and Claire Tomlin and Sergey Levine and Aviral Kumar , booktitle=. What Do Learning Dynamics Reveal About Generalization in. 2025 , url=

    2025

  22. [22]

    arXiv preprint arXiv:2407.10490 , year=

    Learning dynamics of llm finetuning , author=. arXiv preprint arXiv:2407.10490 , year=

    Pith/arXiv arXiv

  23. [23]

    arXiv preprint arXiv:2505.03793 , year=

    LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection , author=. arXiv preprint arXiv:2505.03793 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2407.10671 , year=

    Qwen2 technical report , author=. arXiv preprint arXiv:2407.10671 , year=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  27. [27]

    Journal of Machine Learning Research , volume=

    Scaling instruction-finetuned language models , author=. Journal of Machine Learning Research , volume=

  28. [28]

    Advances in Neural Information Processing Systems , volume=

    Lima: Less is more for alignment , author=. Advances in Neural Information Processing Systems , volume=

  29. [29]

    arXiv preprint arXiv:2109.01652 , year=

    Finetuned language models are zero-shot learners , author=. arXiv preprint arXiv:2109.01652 , year=

    Pith/arXiv arXiv

  30. [30]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  31. [31]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  32. [32]

    arXiv preprint arXiv:2402.14740 , year=

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms , author=. arXiv preprint arXiv:2402.14740 , year=

    Pith/arXiv arXiv

  33. [33]

    2025 , eprint=

    OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs , author=. 2025 , eprint=

    2025

  34. [34]

    OpenMathInstruct-2: Accelerating

    Shubham Toshniwal and Wei Du and Ivan Moshkov and Branislav Kisacanin and Alexan Ayrapetyan and Igor Gitman , booktitle=. OpenMathInstruct-2: Accelerating. 2025 , url=

    2025

  35. [35]

    2024 , eprint=

    Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities , author=. 2024 , eprint=

    2024

  36. [36]

    Forty-first International Conference on Machine Learning , year=

    Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch , author=. Forty-first International Conference on Machine Learning , year=

  37. [37]

    2023 , eprint=

    Editing Models with Task Arithmetic , author=. 2023 , eprint=

    2023

  38. [38]

    Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models , url =

    Ortiz-Jimenez, Guillermo and Favero, Alessandro and Frossard, Pascal , booktitle =. Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models , url =

  39. [39]

    2025 , eprint=

    Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model , author=. 2025 , eprint=

    2025

  40. [40]

    2025 , eprint=

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author=. 2025 , eprint=

    2025

  41. [41]

    2025 , eprint=

    Tulu 3: Pushing Frontiers in Open Language Model Post-Training , author=. 2025 , eprint=

    2025

  42. [42]

    2025 , eprint=

    Rank1: Test-Time Compute for Reranking in Information Retrieval , author=. 2025 , eprint=

    2025

  43. [43]

    2022 , eprint=

    Fusing finetuned models for better pretraining , author=. 2022 , eprint=

    2022

  44. [44]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2103.03874 , year=

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    Pith/arXiv arXiv

  46. [46]

    The Twelfth International Conference on Learning Representations , year=

    Let's verify step by step , author=. The Twelfth International Conference on Learning Representations , year=

  47. [47]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  48. [48]

    arXiv preprint arXiv:2108.07732 , year=

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    Pith/arXiv arXiv

  49. [49]

    arXiv preprint arXiv:2406.19314 , volume=

    Livebench: A challenging, contamination-free llm benchmark , author=. arXiv preprint arXiv:2406.19314 , volume=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2311.07911 , year=

    Instruction-following evaluation for large language models , author=. arXiv preprint arXiv:2311.07911 , year=

    Pith/arXiv arXiv

  51. [51]

    arXiv preprint arXiv:2410.23123 , year=

    On memorization of large language models in logical reasoning , author=. arXiv preprint arXiv:2410.23123 , year=

    Pith/arXiv arXiv

  52. [52]

    Cognition , volume=

    Meta-logical problems: Knights, knaves, and Rips , author=. Cognition , volume=. 1990 , publisher=

    1990

  53. [53]

    arXiv preprint arXiv:2309.15698 , year=

    Deep model fusion: A survey , author=. arXiv preprint arXiv:2309.15698 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2502.18036 , year=

    Harnessing multiple large language models: A survey on llm ensemble , author=. arXiv preprint arXiv:2502.18036 , year=

    Pith/arXiv arXiv

  55. [55]

    arXiv preprint arXiv:2405.11143 , year=

    OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework , author=. arXiv preprint arXiv:2405.11143 , year=

    Pith/arXiv arXiv

  56. [56]

    2024 , journal =

    HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal =

    2024

  57. [57]

    SIAM Journal on Control and Optimization , volume=

    Global convergence of policy gradient methods to (almost) locally optimal policies , author=. SIAM Journal on Control and Optimization , volume=. 2020 , publisher=

    2020

  58. [58]

    Machine learning , volume=

    Simple statistical gradient-following algorithms for connectionist reinforcement learning , author=. Machine learning , volume=. 1992 , publisher=

    1992

  59. [59]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  60. [60]

    arXiv preprint arXiv:2505.11711 , year=

    Reinforcement Learning Finetunes Small Subnetworks in Large Language Models , author=. arXiv preprint arXiv:2505.11711 , year=

    arXiv

  61. [61]

    arXiv preprint arXiv:2509.04259 , year=

    RL's Razor: Why Online Reinforcement Learning Forgets Less , author=. arXiv preprint arXiv:2509.04259 , year=

    Pith/arXiv arXiv

  62. [62]

    arXiv preprint arXiv:2407.06089 , year=

    Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models , author=. arXiv preprint arXiv:2407.06089 , year=

    Pith/arXiv arXiv

  63. [63]

    arXiv preprint arXiv:2503.08998 , year=

    From Task-Specific Models to Unified Systems: A Review of Model Merging Approaches , author=. arXiv preprint arXiv:2503.08998 , year=

    Pith/arXiv arXiv

  64. [64]

    IEEE Transactions on Audio, Speech and Language Processing , volume=

    Imitate, Reward and Introspect: Enhancing Neural Machine Translation by Utilizing Large Language Models as Environments , author=. IEEE Transactions on Audio, Speech and Language Processing , volume=. 2025 , publisher=

    2025

  65. [65]

    arXiv preprint arXiv:2507.08425 , year=

    A survey of large language models in discipline-specific research: Challenges, methods and opportunities , author=. arXiv preprint arXiv:2507.08425 , year=

    Pith/arXiv arXiv

  66. [66]

    arXiv preprint arXiv:2510.02272 , year=

    Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective , author=. arXiv preprint arXiv:2510.02272 , year=

    arXiv

  67. [67]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    X-Instruction: Aligning language model in low-resource languages with self-curated cross-lingual instructions , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024