Pith. sign in

REVIEW 28 cited by

SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2506.19767 v1 pith:2LVCV5IP submitted 2025-06-24 cs.CL cs.AIcs.LG

Yuqian Fu , Tinghong Chen , Jiajun Chai , Xihuai Wang , Songjun Tu , Guojun Yin , Wei Lin , Qichao Zhang
show 2 more authors
Yuanheng Zhu Dongbin Zhao
This is my paper
classification cs.CLcs.AIcs.LG
keywords fine-tuningreasoningreinforcementsrftsupervisedbenchmarksdistributionsintegration
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) have achieved remarkable progress in reasoning tasks, yet the optimal integration of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) remains a fundamental challenge. Through comprehensive analysis of token distributions, learning dynamics, and integration mechanisms from entropy-based perspectives, we reveal key differences between these paradigms: SFT induces coarse-grained global changes to LLM policy distributions, while RL performs fine-grained selective optimizations, with entropy serving as a critical indicator of training effectiveness. Building on these observations, we propose Supervised Reinforcement Fine-Tuning (SRFT), a single-stage method that unifies both fine-tuning paradigms through entropy-aware weighting mechanisms. Our approach simultaneously applies SFT and RL to directly optimize the LLM using demonstrations and self-exploration rollouts rather than through two-stage sequential methods. Extensive experiments show that SRFT achieves 59.1% average accuracy, outperforming zero-RL methods by 9.0% on five mathematical reasoning benchmarks and 10.9% on three out-of-distribution benchmarks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 28 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Hiding future trajectory information until after a driving model forms its decision reduces rationalization and improves verifiable autonomous-driving reasoning in the proposed AD-MCQ and DEFT-RLVR framework.

  2. Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoning

    cs.CV 2026-07 unverdicted novelty 7.0 of 10

    P2R decouples perception from reasoning in VLMs via a two-stage process and PRA-GRPO alternating RL training, reporting gains such as 93.2% on V-Star for the 4B model over its Qwen3-VL backbone.

  3. Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    SGPO extracts strategies from strong-model responses, builds autonomous and guided trajectories, and applies token-level forward-KL distillation with adaptive weighting to outperform SFT and RL baselines by 2.2 points...

  4. Learning Agentic Policy from Action Guidance

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    ActGuide-RL uses human action data as plan-style guidance in mixed-policy RL to overcome exploration barriers in LLM agents, matching SFT+RL performance on search benchmarks without cold-start training.

  5. Near-Future Policy Optimization

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    NPO uses a policy's own near-future checkpoint as auxiliary trajectories to maximize effective learning signal S = Q/V, improving performance from 57.88 to 63.15 on Qwen3-VL-8B-Instruct with GRPO while accelerating co...

  6. $\pi$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data

    cs.LG 2026-04 conditional novelty 7.0 of 10

    Self-play’s natural question-construction paths supply free privileged information that turns sparse-reward self-play into dense self-distillation, yielding stronger data-free search agents.

  7. What are Key Factors for Updates in RL for LLM Reasoning?

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Theoretical analysis of RLVR update dynamics leads to ACPO, an adaptive clipping method that outperforms DAPO and CISPO on reasoning benchmarks with 3B and 7B models.

  8. AIPO: Learning to Reason from Active Interaction

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    AIPO adds active multi-agent consultation (Verify, Knowledge, Reasoning agents) plus custom importance sampling to RLVR training so LLMs expand their reasoning boundary and then operate without the agents.

  9. AIPO: Learning to Reason from Active Interaction

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    AIPO trains LLMs to expand their reasoning capability boundary via active multi-agent interaction with Verify, Knowledge, and Reasoning agents during RLVR, using importance sampling and clipping to handle feedback, th...

  10. Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectors

    cs.LG 2026-05 conditional novelty 6.0 of 10

    DoTS decouples SFT and RLVR training then synthesizes their task vectors at inference time to match integrated training results at ~3% compute cost.

  11. $\pi$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    π-Play uses self-generated question construction paths as privileged information in multi-agent self-distillation to convert sparse-reward self-play into a dense-feedback loop, surpassing supervised search agents and ...

  12. Hindsight-Anchored Policy Optimization: Turning Failure into Feedback in Sparse Reward Settings

    cs.LG 2026-03 unverdicted novelty 6.0 of 10

    HAPO adds a hindsight-anchored SSI operator with Thompson gating to GRPO-style RLVR, achieving asymptotic consistency that recovers unbiased on-policy gradients as the policy improves.

  13. Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models

    cs.LG 2026-02 conditional novelty 6.0 of 10

    CurioSFT preserves exploration during supervised fine-tuning by distilling toward the model's own temperature-scaled distribution and adaptively increasing entropy at high-entropy tokens, improving SFT accuracy by ~2....

  14. Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    A dynamic replay and reweighting scheduler (RECAP) preserves general capabilities during RLVR while keeping reasoning performance at least as good as reasoning-only finetuning.

  15. Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts

    cs.LG 2025-10 conditional novelty 6.0 of 10

    HINT boosts LLM reasoning RL by injecting teacher-generated heuristic hints only on all-failed rollouts, keeping hints out of the policy-optimization prompt, and monitoring guidance quality with a new Affinity metric.

  16. CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems

    cs.MA 2025-09 conditional novelty 6.0 of 10

    CORRECT distills recurring multi-agent failure patterns into reusable error schemas and retrieves them at inference time to localize the decisive error step more accurately than judging or fine-tuning baselines.

  17. EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning

    cs.LG 2025-08 unverdicted novelty 6.0 of 10

    EvoCoT uses self-generated and verified CoT trajectories in a two-stage curriculum to let LLMs learn from initially unsolved hard problems in RLVR settings.

  18. ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Injecting correct reference-policy samples and optimizing a mixed importance-sampling ratio prevents validation collapse and raises asymptotic math-reasoning scores beyond reverse-KL baselines.

  19. ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

    cs.LG 2026-07 conditional novelty 5.0 of 10

    ARMOR adds correct reference-policy anchor samples to each RL batch and optimizes a reference-mixture importance ratio, preventing validation collapse and extending performance gains.

  20. GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    GAC derives adaptive mixing weights for SFT-RL hybrid post-training from online gradient variance and signal disagreement estimates, improving benchmark performance over fixed schedules with under 1% overhead.

  21. LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    LANG combines language-adaptive hint guidance, progressive decay, and difficulty-tailored learning horizons in RL to boost non-English reasoning performance while preserving language consistency.

  22. Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning

    cs.LG 2025-12 unverdicted novelty 5.0 of 10

    Sequential SFT followed by RL, guided by the Plasticity-Ceiling Framework, achieves higher performance ceilings in LLM mathematical reasoning than synchronized methods by optimizing data scale and transition timing.

  23. RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning

    cs.AI 2025-10 conditional novelty 5.0 of 10

    A 3B VLM trained with SFT plus GRPO and an LCS-based reward reaches 55.3% on EmbodiedBench's EB-ALFRED, beating GPT-4o-mini and the 7B REBP planner.

  24. Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    DARS adaptively increases rollouts on hard problems in RLVR to improve Pass@K, and when paired with batch scaling for breadth, achieves gains in both Pass@K and Pass@1 by treating depth and breadth as complementary ex...

  25. Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

    cs.LG 2025-08 reject novelty 5.0 of 10

    DED trains a 32B reasoning model to state-of-the-art levels on AIME and LiveCodeBench using roughly 0.8k curated examples, via teacher selection, hard-example compression, and trajectory diversity.

  26. Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models

    cs.AI 2025-03 unverdicted novelty 5.0 of 10

    The paper unifies perspectives on Long CoT in reasoning LLMs by introducing a taxonomy, detailing characteristics of deep reasoning and reflection, and discussing emergence phenomena and future directions.

  27. Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Models

    cs.AI 2026-05 unverdicted novelty 4.0 of 10

    EKSFT masks high-entropy or high-KL tokens in low-data SFT to preserve pre-trained distribution and improve downstream RL performance on math reasoning tasks.

  28. A Survey of Reinforcement Learning for Large Reasoning Models

    cs.CL 2025-09 accept novelty 3.0 of 10

    A survey compiling RL methods, challenges, data resources, and applications for enhancing reasoning in large language models and large reasoning models since DeepSeek-R1.

Pith tools