Pith. sign in

REVIEW 4 major objections 5 minor 91 references

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-task RL produces nearly orthogonal parameter updates across tasks, while SFT produces dense overlapping ones, so RL can be split into parallel single-task runs whose updates are merged.

desk verdict The empirical recipe for Parallel-RL is genuinely useful; the theoretical explanation is mostly a tautology dressed up as a mechanism, so read the paper for the tables and the trade-off study, not for the proof. read the letter →

arxiv 2608.03573 v2 pith:MYCZXL5M submitted 2026-08-04 cs.CL cs.LG

Kejian Zhu , Zhuoran Jin , Shangqing Tu , Hongbang Yuan , Yushi Bai , Kang Liu , Juanzi Li , Jun Zhao This is my paper · ORCID
classification cs.CLcs.LG
keywords multi-tasklearningsupervisedfine-tuningreinforcementgradientinterferenceparameterorthogonalitytaskcoexistenceLLMreasoningparalleltraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish why supervised fine-tuning (SFT) fails when reasoning tasks are trained one after another, while reinforcement learning (RL) succeeds, and to turn that difference into a practical training scheme. On four reasoning domains, multi-stage SFT lowers average accuracy by 23.1% relative to the base model, whereas multi-stage RL raises it by 24.9%. The proposed explanation is geometric: SFT updates are dense, large, and largely aligned across tasks, while RL updates are sparse and nearly orthogonal, with pairwise cosine similarity around $10^{-5}$. The central theoretical claim is that SFT interference is norm-limited while RL interference is variance-limited, bounded by the variance of within-group rollouts rather than by gradient size. If correct, multi-task RL can be decoupled into parallel single-task runs whose updates are merged, saving training time and allowing modular composition of abilities.

What carries the argument

The load-bearing object is the gradient interference term, the expected inner product between update directions from two different tasks, where a score function is the log-likelihood gradient of a sampled or expert response. For SFT this is $|I_{\mathrm{SFT}}(i,j)| \le M_i M_j$, where $M_i$ bounds the expected norm of the expert score function; for RL the equivalent is $|I_{\mathrm{RL}}(i,j)| \le V_i V_j$, where $V_i$ bounds the intra-group variance of residual score functions. The advantage term in RL, standardized across sampled rollouts, has zero sum, so the common direction $\bar{S}$ cancels and only residuals $\delta S$ remain; concentration of measure then makes independent, zero-mean, sparse residual vectors nearly orthogonal in the high-dimensional parameter space. This is what lets gradient interference in RL be variance-limited rather than norm-limited, and it is the mechanism that Parallel-RL exploits by merging independently trained task updates.

What would settle it

One concrete falsifier: take two deliberately similar tasks, estimate $V_i$ and $V_j$ on large rollout batches, and measure the cross-task cosine similarity of the full RL updates; if $V_i V_j$ is non-negligible or the cosine is orders of magnitude above the reported $10^{-3}$ level, the variance-limited orthogonality mechanism is not operating.

Watch

Extended reading notes

Core claim

The paper's central claim is that the conflict between tasks in multi-task training is determined by the geometry of parameter updates. In SFT, each task's gradient points along a dense, high-magnitude direction, and these directions overlap strongly across tasks, so interference grows with the absolute gradient norm. In RL with group-normalized advantages, the zero-sum property of the advantage weights algebraically cancels the common mean direction inside each group, leaving only residual score vectors that are small, sparse, and approximately orthogonal across tasks, so interference is bounded by the intra-group variance of those residuals. This variance-limited bound is what the paper means by "RL coexists," and it is the reason multi-stage RL accumulates gains. The paper then proposes Parallel-RL: train each task independently in parallel and merge the updates, reporting that naive summation retains most single-task gains and that a light post-merge adaptation can slightly exceed single-task performance.

Load-bearing premise

The argument's load-bearing premise is that the residual score vectors left after subtracting each rollout group's mean are independent, zero-mean, sparse vectors with genuinely small variance; if that fails, the orthogonality and the Parallel-RL benefit do not follow from the mathematics.

Editorial extensions

If this is right

  • Multi-stage SFT on diverse reasoning tasks should be avoided, since its dense overlapping updates cause large cross-task interference and collapse.
  • Multi-task RL can be run as independent single-task trainings whose updates are summed or averaged, with near-single-task retention of roughly 95% to 98% in the reported setups.
  • Because RL interference is variance-limited, raising rollout diversity, for example by increasing sampling temperature, also raises cross-task interference, so exploration must be balanced against merge quality.
  • Task pairs with overlapping sampled score-function distributions are not safely mergeable; the paper's logic-game counterexample shows such pairs lose performance in Parallel-RL.
  • A short post-merge RL adaptation on a small amount of data can recover and even exceed single-task performance, so decoupling does not force a final-quality penalty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not develop is that the variance-limited bound predicts a ceiling on exploration temperature in parallel post-training; at sufficiently high temperature the merged model should lose the additive gains even though each single-task model improves.
  • The paper's task-compatibility discussion suggests a quantitative pre-training filter: measure whether the sampled score-function distributions of two candidate tasks separate, rather than relying on visual t-SNE inspection.
  • By the same variance-limited logic, algorithms that normalize advantages over a whole batch rather than per prompt group should interfere more for the same data; the paper's PPO comparison is consistent with this, though a general law is our inference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports an empirical and theoretical study of multi-task post-training for large language models, contrasting Supervised Fine-Tuning (SFT) with reinforcement learning (RL), primarily GRPO. The authors report that multi-stage SFT degrades performance sharply, while multi-stage RL yields cumulative gains; single-task RL also preserves or improves performance on untrained tasks, unlike SFT. At the parameter level, they observe that RL updates have smaller norm, higher sparsity, and lower cross-task cosine similarity than SFT updates. The theory proposes that SFT gradient interference is norm-limited (bounded by M_i M_j) whereas RL interference is variance-limited (bounded by V_i V_j), based on a decomposition of GRPO gradients into residual score functions. The paper then introduces Parallel-RL, which trains tasks independently in parallel and merges the parameter updates; experiments with GRPO and PPO show that such merging retains most single-task gains.

Significance. If the central claims were fully established, the paper would give a mechanistic explanation of why RL supports multi-task training while SFT does not, and it would justify a practical decoupling paradigm, Parallel-RL, with meaningful efficiency benefits. The empirical component is substantial: it spans two model sizes, full-parameter and LoRA training, GRPO and PPO, a range of benchmarks, and extensive ablations including a task-compatibility analysis. These empirical findings are valuable and likely reproducible. However, the theoretical explanation, as presented, is not yet load-bearing: the main bound restates an assumption, the concentration-of-measure argument is invoked without its required conditions, and the step from per-step gradient bounds to the orthogonality of final parameter updates is missing. The paper's current value rests on its empirical contributions and on the practical demonstration of Parallel-RL, not on the advertised theoretical derivation.

major comments (4)
  1. [Section 4.3, Theorem 4.5 and Assumption 4.4] The bound |I_RL| ≤ V_i V_j is essentially the content of Assumption 4.4 combined with Cauchy-Schwarz; the proof's intermediate steps do not derive smallness of V_i from on-policy structure or advantage normalization. Section 4.4 measures ||δS|| ≈ 10^{-2}, so the 'variance-limited' conclusion is an observed empirical property, not a theoretical prediction. To make the theorem explanatory, the authors need an argument that V_i is small (e.g., from a KL constraint or a bound on intra-group policy divergence), or they should explicitly reframe V_i as an empirically calibrated constant rather than a derived quantity.
  2. [Section 4.2, after Eq. (7)] The concentration-of-measure bound P(|⟨δS_i,δS_j⟩| ≥ t) ≤ 2 exp(−ct²d) is invoked for 'independent, zero-mean and sparse' vectors without stating the required conditions. Sparsity alone does not imply near-orthogonality unless the nonzero supports are random or nearly disjoint; the paper does not rule out the realistic case where all task residuals share a common support or occupy a common low-dimensional subspace. Moreover, the residuals within a GRPO group are not independent (they sum to zero by construction). Thus the claimed near-orthogonality of cross-task gradients is an additional assumption, not a consequence of concentration of measure.
  3. [Section 5.1 and Section 4.3] Theorem 4.5 bounds the expected inner product of single-step gradients (Eq. 6), while the empirical claim in Section 3 and the Parallel-RL merging procedure concern final parameter updates ΔW_i = Σ_t η_t g_{i,t}. The interference term ⟨ΔW_i,ΔW_j⟩ contains T² cross-time terms Σ_{t,s} ⟨g_{i,t},g_{j,s}⟩, and the proof provides no control of the off-diagonal t≠s terms. The statement in Section 5.1 that 'optimization directions are irrelevant' is exactly the unproved bridge from per-step orthogonality to final-update orthogonality. The empirical success of Parallel-RL in Table 4 is credible, but the stated theoretical justification is incomplete.
  4. [Section 3, Observation 2 and Figure 2] The text states that the average pairwise cosine similarity between RL updates is 'around 10^{-5}', but the six off-diagonal entries shown in the RL heatmap are on the order of 10^{-4} (e.g., 4.49×10^{-4}, −4.83×10^{-4}); the average absolute value is approximately 2.3×10^{-4}. This numerical inconsistency weakens the quantitative support for the claimed orthogonality and should be corrected.
minor comments (5)
  1. [Abstract] The abstract contains a stray token '/githubCode' at the end; this appears to be a broken link or formatting artifact and should be replaced with the actual code repository URL or removed.
  2. [Table 3] The formatting of Table 3 is garbled: the row '||S||2 ∼7.1∼10−1' does not clearly separate the SFT and RL columns, making the reported magnitudes difficult to read. Please reformat the table with explicit column headers.
  3. [Tables 2 and 6] The meaning of the superscripts is inconsistent across tables: in Table 1 and Table 2 they denote changes relative to the Base Model, while in Table 6 they denote gaps relative to the Single-Task baseline. These conventions should be unified and clearly stated in every caption.
  4. [Section 4.2] The phrase 'Similarity, we qualitatively examine' should read 'Similarly, we qualitatively examine.' Please also check for other typographical errors in the same paragraph, where 'independent zero-mean vectors' should probably be 'independent, zero-mean vectors.'
  5. [Proposition 4.1] Proposition 4.1 is quoted from Shenfeld et al. under conditions (binary rewards, convex policy set) and is used to explain sparsity of RL updates, but the experiments use GRPO with non-binary rewards and the KL penalty is explicitly disabled in Appendix A.2. The authors should clarify how Proposition 4.1 applies to their setting or state that the sparsity explanation is empirical.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central bound is a theorem from stated assumptions; the smallness of the RL variance is a measured input, and the main weakness is an unsupported inference from per-step gradients to accumulated updates, not a definitional reduction.

full rationale

The core derivation chain in Section 4 is not circular. I_RL is defined independently in Eq. (6) as the expected inner product of per-step GRPO gradients; Lemma 4.3 uses the zero-sum property of standardized advantages to replace full score functions by residuals δS, an algebraic step with genuine content. Theorem 4.5 then derives |I_RL| ≤ V_i V_j from Jensen and Cauchy-Schwarz under Assumption 4.4. The theorem is not a restatement of the assumption: the assumption bounds the residual variance, while the conclusion bounds a different quantity, the cross-task gradient inner product. The smallness of V_i is empirically measured in Section 4.4 (Table 3) rather than fitted to reproduce the observed interference, so this is a bound with a measured constant rather than a fitted parameter renamed as a prediction. The sparsity premise is supported by an external result (Shenfeld et al. 2025) and standard concentration arguments (Vershynin 2018), and the Parallel-RL application is tested against held-out benchmarks, providing independent empirical content. The genuine weaknesses are inferential gaps, not circularity: Theorem 4.5 bounds per-step gradient inner products, while Section 5.1 concludes that ⟨ΔW_i, ΔW_j⟩ is negligible for accumulated updates, a step requiring an unstated temporal independence or anti-correlation argument; and Section 4.2 asserts rather than derives that intra-group residual variance is small. These are missing support, not equivalence of inputs and outputs by construction, so the paper does not warrant a circularity finding under the stated criteria.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central theoretical claims rest on measured quantities (M_i, V_i), an unverified concentration-of-measure application, and several domain assumptions about task independence and small intra-group variance. There are no new physical or conceptual entities beyond the Parallel-RL method itself.

free parameters (3)
  • V_i (RL residual variance bound) = empirically ~1e-2 (Table 3), not predicted
    The variance-limited interference bound is only informative if V_i is small; the paper measures rather than derives this smallness, so the bound's explanatory content rests on this observed quantity.
  • M_i (SFT score norm bound) = empirically ~7.1 (Table 3), not predicted
    The norm-limited SFT bound is meaningful only once M_i is measured; the theory does not predict it.
  • Sparsity threshold (1e-5) = 1e-5
    Observation 1 defines 'sparse' as parameters with magnitude exceeding 1e-5; this threshold is chosen by hand and affects the reported 20% vs 93% sparsity comparison.
assumptions (5)
  • domain assumption Assumption 4.4: expected squared norm of SFT score functions bounded by M_i^2 and expected intra-group variance of RL residual score functions bounded by V_i^2
    Theorem 4.5 is a direct consequence of these bounds; their smallness for RL is an empirical input, not a consequence of the theory.
  • ad hoc to paper Residual score vectors for different tasks are independent, zero-mean, and sparse, so high-dimensional concentration implies near-orthogonality (Section 4.2)
    Zero-mean and independence are asserted, not proven; actual residuals are defined relative to group means and may have nonzero expectation across inputs.
  • ad hoc to paper Intra-group policy divergence is inherently limited because model and input are fixed during rollout generation (Section 4.2)
    No formal bound is provided; this is the key assumption that makes V_i small and is justified only by the measured value.
  • domain assumption Data distributions for different tasks are independent (Section 4.2)
    Used to factor expectations, but in practice tasks come from the same base corpus and may be correlated.
  • standard math Proposition 4.1 (Shenfeld et al.): policy gradient converges to the KL-minimal optimal policy under suitable regularity conditions
    Cited from prior work; used to argue RL updates are small. Assumes a convex policy family and binary reward, which may not hold exactly for LLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs." pith.science (2026). https://pith.science/paper/MYCZXL5M

@misc{pith2026260803573,
  author       = {Pith},
  title        = {Pith review of: SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MYCZXL5M}},
  note         = {Machine review of arXiv:2608.03573}
}
read the original abstract

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.

Figures

Figures reproduced from arXiv: 2608.03573 by the authors.

Figure 1
Figure 1. SFT and RL exhibit significant differences during multi￾stage training. SFT shows task conflicts, which impairs perfor￾mance, while RL can steadily improve performance across tasks. for achieving Artificial General Intelligence (AGI), remain underexplored. We observe that in most existing works, researchers typically construct mixed datasets from multiple tasks for SFT (Dong et al., 2023; Park, 2025). In contrast, f… view at source ↗
Figure 2
Figure 2. Analysis of Parameter Update Dynamics across SFT and RL. The heatmaps display the pairwise cosine similarity (off-diagonal) and the L2 norm magnitude (diagonal) of parameter update vectors for different tasks. Left: SFT update has a larger absolute similarity value. (≈ 1.0). Right: RL updates show near-zero similarity, indicating orthogonal and task-specific parameter adjustments. gSFT = Ex∼D,y∼πexpert [∇θ log πθ(y|… view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Analysis of RL training on interfering tasks. Top: Distribution of the score function S sampled during RL training on math, code and game. Bottom: Corresponding performance comparison on Math (AIME25) and Code (LiveCodeBench) tasks. Training game models in parallel wit…
Figure 5
Figure 5. Figure 5: The Trade-off between Single-Task Exploration and Multi-Task Coexis￾tence. When rollout temperature τ grows, we calculates the performance of Math Model, intra-group gradient variance (measured by − log V ), and Naive Parallel￾RL accuracy on MATH500. Increasing τ impro…
Figure 6
Figure 6. Figure 6: Distribution of the score function S sampled during training by RL and SFT on different tasks (Math, Code and Logic). We use t-SNE to reduce the dimensionality of the high-dimensional score function vectors for visualization. E.3. More Visualizations In Section 4.4, we…
Figure 7
Figure 7. Figure 7: Pass@K Analysis of Naive Parallel RL with full parameter GRPO As observed in the figures, although the Parallel-RL model may exhibit a performance drop at Pass@1 compared to the task-specific RL models, the Pass@k curves of the two models tend to converge as the number…
Figure 8
Figure 8. Figure 8: Response from the base model (DeepSeek-R1-Distill-Qwen-1.5B) on a sample problem from MATH500. As shown, the base model fails to solve the problem. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]
Figure 9
Figure 9. Figure 9: Response from the logic model on a sample problem from MATH500. The logic model is derived from the base model via GRPO training on a logic dataset. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_9.png]
Figure 10
Figure 10. Figure 10: Response from the math model on a sample problem from MATH500. The math model is derived from the base model via GRPO training on a math dataset. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]
Figure 11
Figure 11. Figure 11: Response from the Parallel-RL model on a sample problem from MATH500. This model uses Naive Parallel-RL (shown in Section 5.2), which is obtained by directly averaging the models trained on four tasks. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_11.png]
Figure 12
Figure 12. Figure 12: Comparison of responses from the base model (DeepSeek-R1-Distill-Qwen-1.5B) and the logic model on a sample problem from Knights and Knaves. The logic model is derived from the base model via GRPO training on a logic dataset. As shown, the base model fails to solve th…
Figure 13
Figure 13. Figure 13: Response from the math model on a sample problem from Knights and Knaves. The math model is derived from the base model via GRPO training on a math dataset. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_13.png]
Figure 14
Figure 14. Figure 14: Response from the Parallel-RL model on a sample problem from Knights and Knaves. This model uses Naive Parallel-RL (shown in Section 5.2), which is obtained by directly averaging the models trained on four tasks. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_14.png]
Figure 15
Figure 15. Figure 15: Prompt for Math and Science Task Training. We follow the prompt template settings in deepscaler. Prompt for AIME Evaluation <|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user {question} Solve the following math problem efficiently and clearly. …
Figure 16
Figure 16. Figure 16: Prompt for AIME Evaluation. We follow the prompt template settings in Lighteval. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_16.png]
Figure 17
Figure 17. Figure 17: Prompt for MATH500 Evaluation. We follow the prompt template settings in Lighteval. Prompt for GPQA Evaluation <|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user Answer the following multiple choice question. The last line of your response shou…
Figure 18
Figure 18. Figure 18: Prompt for GPQA Evaluation. We follow the prompt template settings in Lighteval. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_18.png]
Figure 19
Figure 19. Figure 19: Prompt for MMLU Evaluation. We follow the prompt template settings in Lighteval. Prompt for Logic Task (Knights&Knaves) <|im_start|>system You are a helpful assistant. The assistant first thinks about the reasoning process in the mind and then provides the user with t…
Figure 20
Figure 20. Figure 20: Prompt for Logic Task Training and Evaluation. We follow the prompt template settings in LogicRL. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_20.png]
Figure 21
Figure 21. Figure 21: Prompt for Code Task Training and Evaluation. We follow the prompt template settings in LiveCodeBench and deepcoder. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_21.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 21 canonical work pages

  1. [1]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  2. [2]

    2025 , eprint=

    Imbalanced Gradients in RL Post-Training of Multi-Task LLMs , author=. 2025 , eprint=

    2025

  3. [3]

    Notion Blog , year=

    Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl , author=. Notion Blog , year=

  4. [4]

    Notion Blog , year=

    Deepcoder: A fully open-source 14b coder at o3-mini level , author=. Notion Blog , year=

  5. [5]

    arXiv preprint arXiv:2502.14768 , year=

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning , author=. arXiv preprint arXiv:2502.14768 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2508.06471 , year=

    Glm-4.5: Agentic, reasoning, and coding (arc) foundation models , author=. arXiv preprint arXiv:2508.06471 , year=

    arXiv

  7. [7]

    2025 , eprint=

    Not All Correct Answers Are Equal: Why Your Distillation Source Matters , author=. 2025 , eprint=

    2025

  8. [8]

    arXiv preprint arXiv:2501.17161 , year=

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training , author=. arXiv preprint arXiv:2501.17161 , year=

    arXiv

Show all 91 references
  1. [9]

    arXiv preprint arXiv:2509.21128 , year=

    Rl squeezes, sft expands: A comparative study of reasoning llms , author=. arXiv preprint arXiv:2509.21128 , year=

    arXiv

  2. [10]

    arXiv preprint arXiv:2310.05492 , year=

    How abilities in large language models are affected by supervised fine-tuning data composition , author=. arXiv preprint arXiv:2310.05492 , year=

    arXiv

  3. [11]

    2025 , school=

    Instruct-skillmix: A powerful pipeline for llm instruction tuning , author=. 2025 , school=

    2025

  4. [12]

    Journal of Machine Learning Research , volume=

    Curriculum learning for reinforcement learning domains: A framework and survey , author=. Journal of Machine Learning Research , volume=

  5. [13]

    Forty-first International Conference on Machine Learning , year=

    Hard tasks first: Multi-task reinforcement learning through task scheduling , author=. Forty-first International Conference on Machine Learning , year=

  6. [14]

    2024 , journal =

    HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal =

    2024

  7. [15]

    arXiv preprint arXiv:2403.13372 , year=

    Llamafactory: Unified efficient fine-tuning of 100+ language models , author=. arXiv preprint arXiv:2403.13372 , year=

    arXiv

  8. [16]

    arXiv preprint arXiv:2505.11711 , year=

    Reinforcement Learning Finetunes Small Subnetworks in Large Language Models , author=. arXiv preprint arXiv:2505.11711 , year=

  9. [17]

    arXiv preprint arXiv:2412.16720 , year=

    Openai o1 system card , author=. arXiv preprint arXiv:2412.16720 , year=

    arXiv

  10. [18]

    arXiv preprint arXiv:2510.00553 , year=

    On Predictability of Reinforcement Learning Dynamics for Large Language Models , author=. arXiv preprint arXiv:2510.00553 , year=

  11. [19]

    Advances in Neural Information Processing Systems , volume=

    Ties-merging: Resolving interference when merging models , author=. Advances in Neural Information Processing Systems , volume=

  12. [20]

    arXiv preprint arXiv:2509.04259 , year=

    RL's Razor: Why Online Reinforcement Learning Forgets Less , author=. arXiv preprint arXiv:2509.04259 , year=

    arXiv

  13. [21]

    2025 , eprint=

    Reinforcement Learning Finetunes Small Subnetworks in Large Language Models , author=. 2025 , eprint=

    2025

  14. [22]

    2013 , eprint=

    Distributions of Angles in Random Packing on Spheres , author=. 2013 , eprint=

    2013

  15. [23]

    2025 , eprint=

    A short survey on almost orthogonal vectors in a few specific large dimensions , author=. 2025 , eprint=

    2025

  16. [24]

    Advances in Neural Information Processing Systems , volume=

    How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model , author=. Advances in Neural Information Processing Systems , volume=

  17. [25]

    arXiv preprint arXiv:2211.07349 , year=

    Finding skill neurons in pre-trained transformer-based language models , author=. arXiv preprint arXiv:2211.07349 , year=

    arXiv

  18. [26]

    arXiv preprint arXiv:2501.11284 , year=

    Redstar: Does scaling long-cot data unlock better slow-reasoning systems? , author=. arXiv preprint arXiv:2501.11284 , year=

    arXiv

  19. [27]

    Open R1: A fully open reproduction of DeepSeek-R1 , url =

  20. [28]

    arXiv preprint arXiv:2507.18071 , year=

    Group sequence policy optimization , author=. arXiv preprint arXiv:2507.18071 , year=

    arXiv

  21. [29]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  22. [30]

    reinforcement learning fine-tuning for llms , author=

    Rl is neither a panacea nor a mirage: Understanding supervised vs. reinforcement learning fine-tuning for llms , author=. arXiv preprint arXiv:2508.16546 , year=

    arXiv

  23. [31]

    AN IMPLICIT COMPASS DRIVES OPTIMIZATION BIAS , author=

  24. [32]

    arXiv preprint arXiv:2508.11408 , year=

    On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting , author=. arXiv preprint arXiv:2508.11408 , year=

  25. [33]

    arXiv preprint arXiv:2506.19767 , year=

    SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning , author=. arXiv preprint arXiv:2506.19767 , year=

    arXiv

  26. [34]

    arXiv preprint arXiv:2507.14783 , year=

    Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling , author=. arXiv preprint arXiv:2507.14783 , year=

  27. [35]

    AComprehensive ANALYSIS OF HOW REINFORCE-MENT LEARNING MITIGATES TASK CONFLICTS IN LLMS , author=

  28. [36]

    arXiv preprint arXiv:2506.04142 , year=

    Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis , author=. arXiv preprint arXiv:2506.04142 , year=

    arXiv

  29. [37]

    2024 , eprint=

    GPT-4 Technical Report , author=. 2024 , eprint=

    2024

  30. [38]

    arXiv e-prints , pages=

    The llama 3 herd of models , author=. arXiv e-prints , pages=

  31. [39]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  32. [40]

    arXiv preprint arXiv:2505.11423 , year=

    When thinking fails: The pitfalls of reasoning for instruction-following in llms , author=. arXiv preprint arXiv:2505.11423 , year=

    arXiv

  33. [41]

    International Conference on Machine Learning , pages=

    The flan collection: Designing data and methods for effective instruction tuning , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  34. [42]

    arXiv preprint arXiv:1909.08593 , year=

    Fine-tuning language models from human preferences , author=. arXiv preprint arXiv:1909.08593 , year=

    1909 arXiv

  35. [43]

    arXiv preprint arXiv:2510.23451 , year=

    Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences , author=. arXiv preprint arXiv:2510.23451 , year=

  36. [44]

    arXiv preprint arXiv:2503.14476 , year=

    Dapo: An open-source llm reinforcement learning system at scale , author=. arXiv preprint arXiv:2503.14476 , year=

    arXiv

  37. [45]

    arXiv preprint arXiv:2507.20534 , year=

    Kimi k2: Open agentic intelligence , author=. arXiv preprint arXiv:2507.20534 , year=

    arXiv

  38. [46]

    arXiv preprint arXiv:2109.01652 , year=

    Finetuned language models are zero-shot learners , author=. arXiv preprint arXiv:2109.01652 , year=

    arXiv

  39. [47]

    arXiv preprint arXiv:2507.00432 , year=

    Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning , author=. arXiv preprint arXiv:2507.00432 , year=

    arXiv

  40. [48]

    2025 , eprint=

    Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training , author=. 2025 , eprint=

    2025

  41. [49]

    arXiv preprint arXiv:2510.19178 , year=

    Imbalanced Gradients in RL Post-Training of Multi-Task LLMs , author=. arXiv preprint arXiv:2510.19178 , year=

  42. [50]

    2: Pushing the frontier of open large language models , author=

    Deepseek-v3. 2: Pushing the frontier of open large language models , author=. arXiv preprint arXiv:2512.02556 , year=

    arXiv

  43. [51]

    arXiv preprint arXiv:2511.07317 , year=

    RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments , author=. arXiv preprint arXiv:2511.07317 , year=

    arXiv

  44. [52]

    2025 , url =

    OpenAI , title =. 2025 , url =

    2025

  45. [53]

    2025 , url =

    Google , title =. 2025 , url =

    2025

  46. [54]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  47. [55]

    arXiv preprint arXiv:2504.17565 , year=

    Deepdistill: Enhancing llm reasoning capabilities via large-scale difficulty-graded data training , author=. arXiv preprint arXiv:2504.17565 , year=

    arXiv

  48. [56]

    2024 , eprint=

    On Memorization of Large Language Models in Logical Reasoning , author=. 2024 , eprint=

    2024

  49. [57]

    2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages=

    Deepscaler: Holistic autoscaling for microservices based on spatiotemporal gnn with adaptive graph learning , author=. 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages=. 2023 , organization=

    2023

  50. [58]

    arXiv preprint arXiv:2305.20050 , year=

    Let's Verify Step by Step , author=. arXiv preprint arXiv:2305.20050 , year=

    arXiv

  51. [59]

    2021 , eprint=

    Measuring Massive Multitask Language Understanding , author=. 2021 , eprint=

    2021

  52. [60]

    2023 , eprint=

    GPQA: A Graduate-Level Google-Proof Q&A Benchmark , author=. 2023 , eprint=

    2023

  53. [61]

    2024 , eprint=

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , author=. 2024 , eprint=

    2024

  54. [62]

    2021 , eprint=

    LoRA: Low-Rank Adaptation of Large Language Models , author=. 2021 , eprint=

    2021

  55. [63]

    2023 , version =

    Habib, Nathan and Fourrier, Clémentine and Kydlíček, Hynek and Wolf, Thomas and Tunstall, Lewis , title =. 2023 , version =

    2023

  56. [64]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    arXiv

  57. [65]

    arXiv preprint arXiv:2505.19641 , year=

    SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond , author=. arXiv preprint arXiv:2505.19641 , year=

    arXiv

  58. [66]

    2025 , eprint=

    Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? , author=. 2025 , eprint=

    2025

  59. [67]

    2021 , eprint=

    Evaluating Large Language Models Trained on Code , author=. 2021 , eprint=

    2021

  60. [68]

    2018 , publisher=

    High-dimensional probability: An introduction with applications in data science , author=. 2018 , publisher=

    2018

  61. [69]

    arXiv preprint arXiv:2511.08567 , year=

    The path not taken: Rlvr provably learns off the principals , author=. arXiv preprint arXiv:2511.08567 , year=

  62. [70]

    2023 , eprint=

    Editing Models with Task Arithmetic , author=. 2023 , eprint=

    2023

  63. [71]

    arXiv preprint arXiv:2507.17512 , year=

    Can one domain help others? a data-centric study on multi-domain reasoning via reinforcement learning , author=. arXiv preprint arXiv:2507.17512 , year=

    arXiv

  64. [72]

    arXiv preprint arXiv:2506.14965 , year=

    Revisiting reinforcement learning for llm reasoning from a cross-domain perspective , author=. arXiv preprint arXiv:2506.14965 , year=

    arXiv

  65. [73]

    The Fourteenth International Conference on Learning Representations , year=

    Boosting multi-domain reasoning of llms via curvature-guided policy optimization , author=. The Fourteenth International Conference on Learning Representations , year=

  66. [74]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Mixture-of-skills: Learning to optimize data usage for fine-tuning large language models , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  67. [75]

    Forty-second International Conference on Machine Learning , year=

    Boosting multi-domain fine-tuning of large language models through evolving interactions between samples , author=. Forty-second International Conference on Machine Learning , year=

  68. [76]

    International conference on machine learning , pages=

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  69. [77]

    arXiv preprint arXiv:2212.04089 , year=

    Editing models with task arithmetic , author=. arXiv preprint arXiv:2212.04089 , year=

    arXiv

  70. [78]

    Forty-first International Conference on Machine Learning , year=

    Language models are super mario: Absorbing abilities from homologous models as a free lunch , author=. Forty-first International Conference on Machine Learning , year=

  71. [79]

    Advances in Neural Information Processing Systems , volume=

    Merging models with fisher-weighted averaging , author=. Advances in Neural Information Processing Systems , volume=

  72. [80]

    arXiv preprint arXiv:2212.09849 , year=

    Dataless knowledge fusion by merging weights of language models , author=. arXiv preprint arXiv:2212.09849 , year=

    arXiv

  73. [81]

    2026 , eprint=

    Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application , author=. 2026 , eprint=

    2026

  74. [82]

    2026 , eprint=

    The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation , author=. 2026 , eprint=

    2026

  75. [83]

    2026 , eprint=

    Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do , author=. 2026 , eprint=

    2026

  76. [84]

    2026 , eprint=

    Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies , author=. 2026 , eprint=

    2026

  77. [85]

    2026 , eprint=

    From P(y|x) to P(y) : Investigating Reinforcement Learning in Pre-train Space , author=. 2026 , eprint=

    2026

  78. [86]

    2026 , eprint=

    Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It , author=. 2026 , eprint=

    2026

  79. [87]

    2026 , eprint=

    MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos , author=. 2026 , eprint=

    2026

  80. [88]

    2026 , eprint=

    MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning , author=. 2026 , eprint=

    2026

  81. [89]

    2026 , eprint=

    Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling , author=. 2026 , eprint=

    2026

  82. [90]

    2025 , eprint=

    Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents , author=. 2025 , eprint=

    2025

  83. [91]

    2026 , eprint=

    Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning , author=. 2026 , eprint=

    2026

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.