Pith. sign in

REVIEW 4 major objections 6 minor 28 references

UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Selecting the 10% of RL training examples nearest the dataset's mean confidence matches or beats full-data training.

desk verdict A useful single-pass data-selection heuristic with a plausible ZPD story, but the headline claims outrun the evidence: one model clearly regresses and the mean-confidence anchor is unjustified. read the letter →

arxiv 2505.12457 v1 pith:MCRSEHQT submitted 2025-05-18 cs.LG cs.CL

Yang Zhao , Kai Xiong , Xiao Ding , Li Du , YangouOuyang , Zhouhao Sun , Jiannan Guan , Wenbin Zhang
show 4 more authors
Bin Liu Dong Hu Bing Qin Ting Liu
This is my paper · ORCID
classification cs.LGcs.CL
keywords reinforcementlearningdataselectionuncertaintyestimationlargelanguagemodelszoneofproximaldevelopmentmathematicalreasoningtrainingefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the most valuable data for reinforcement-learning fine-tuning of large language models are examples of intermediate difficulty: problems the model has not mastered but can plausibly learn. It proposes a single forward-pass confidence score, the average log-probability of the generated tokens, and ranks training examples by how close their confidence is to the dataset mean. Training on the top 10% of examples by this ranking, the paper reports, matches or beats training on the full dataset on mathematical reasoning benchmarks, while cutting data-evaluation cost by up to 185x and total fine-tuning time by up to 16x. If correct, this would make large-scale RL fine-tuning substantially cheaper and more stable. The practical appeal is that the selection rule is cheap enough to run inside a training loop rather than as a separate multi-sample evaluation stage.

What carries the argument

The load-bearing object is the fuzziness score $\mathrm{Score}(s_i) = 1 - (s_i - \mu)^2$, with $s_i = \exp(\mathrm{Conf}(x_i))$ the geometric mean of token probabilities from one decoding pass and $\mu$ the mean of $s_i$ over the candidate dataset. This score operationalizes the Zone of Proximal Development as 'closest to the dataset-average confidence': samples exactly at $\mu$ get score 1, and the top 10% by this score are the training set. The efficiency carrier is the confidence measure itself, $\mathrm{Conf}(x_i)$, which requires a single forward pass and no iterative next-token rollouts, making data evaluation up to 185x faster than 16-sample accuracy estimation while providing a continuous, fine-grained ranking.

What would settle it

On a fresh model and dataset, compute per-decile learning curves as in Figure 1, and then compare the decile containing the dataset-mean confidence with the empirically best decile; if the mean-confidence decile is not among the top bins, or if a selector centered at a different quantile beats $\mathrm{Score}(s_i) = 1 - (s_i - \mu)^2$, the load-bearing assumption is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that RL training value is a non-monotonic function of the model's current uncertainty: data with intermediate uncertainty produce the largest learning gains, while very easy data yield diminishing returns and very hard data destabilize training. The paper then claims that a continuous, single-pass confidence estimate—the average log-probability of output tokens, written $\mathrm{Conf}(x_i) = \frac{1}{T}\sum_t \log P(y_t \mid x_i, y_{<t})$—captures this difficulty ordering well enough to select data, correlating with multi-sample accuracy between 0.68 and 0.86 across six models. On that basis, it ranks examples by $\mathrm{Score}(s_i) = 1 - (s_i - \mu)^2$, where $s_i = \exp(\mathrm{Conf}(x_i))$ and $\mu$ is the mean confidence of the candidate set, and keeps the top 10%. Across GSM8K and DAPO-MATH-17K training runs on models from 0.5B to 8B parameters, the selected 10% matches or exceeds full-data accuracy, improves Math500 generalization in most cases, and cuts total training time by a factor of 11 to 16.

Load-bearing premise

The selection rule assumes that the candidate dataset's mean confidence marks the exact difficulty at which the model learns best, so the highest-value samples are the ones closest to that average.

Editorial extensions

If this is right

  • RL fine-tuning can be run on 10% of the original training data with final accuracy comparable to or better than full-data training on both GSM8K and DAPO-MATH-17K.
  • Data evaluation for selection reduces to one batched forward pass, achieving up to a 185x speedup over estimating difficulty from 16 samples per instance.
  • Total RL training time drops by 11x to 16x, because fewer instances enter the policy loop and the later, longer-rollout training stages are cut short.
  • Training on intermediate-confidence data improves stability, so smaller models avoid the catastrophic performance drops that full-data training on DAPO-MATH-17K can cause.
  • Generalization to unseen math problems (Math500) improves or matches full-data training in most model and dataset combinations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The mean-confidence rule is one instance of a broader curriculum: any quantile of the confidence distribution could be tested as the ZPD center, and a selector that adapts the center as the model trains might outperform the static 10% selection reported here.
  • Because confidence is model-specific and does not require rewards, the same score applies to any token-level generative task; if the non-monotonic difficulty curve holds beyond mathematics, the selection rule becomes a general cheap data-curriculum for RL.
  • The paper's Figure 1 suggests the optimal difficulty bin varies with model size, so the fixed 'dataset mean' center may be a coarse stand-in; locating the optimal bin per model and checking whether it tracks $\mu$ over training would be a direct stress test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces UFO-RL, a data-selection method for reinforcement-learning fine-tuning of LLMs. The authors first present a preliminary study showing that training on data binned by multi-sample sampling accuracy has a non-monotonic effect on downstream accuracy, with intermediate-difficulty data being most beneficial. They then propose replacing expensive multi-sample accuracy with a single-pass confidence score (average log-softmax over generated tokens), define a 'fuzziness score' Score(s_i)=1-(s_i-mu)^2 centered at the mean confidence of the candidate dataset, and select the top 10% of examples by this score for RL training. Experiments on GSM8K and DAPO-MATH-17K across six models (Qwen2.5-0.5B through 7B, Llama-3.1-8B, Mistral-7B) report that this 10% subset yields performance comparable to or better than full-data training, that confidence evaluation is up to 185x faster than multi-sample accuracy evaluation, and that overall RL training time is reduced by up to 16x. The paper also claims improved training stability and generalization.

Significance. If the central claims hold, the paper makes a practically valuable contribution: it offers a cheap, single-pass proxy for example difficulty and shows, across several model scales and two training sets, that a small curated subset can match full-data RL fine-tuning on in-domain and near-domain benchmarks. The 185x reduction in data-evaluation cost is a concrete and useful engineering result, and the ZPD framing provides an interpretable selection principle. The empirical breadth (six models, two training datasets, three evaluation benchmarks) is a real strength. However, the significance is currently undercut by overstatements in the abstract and introduction that are contradicted by the paper's own tables, and by an under-justified operationalization of the ZPD center as the dataset-mean confidence. The paper does not release code or seeds, which limits reproducibility of the main empirical claims.

major comments (4)
  1. [Abstract and Section 5.2, Table 5] The claim that training on 10% of UFO-RL-selected data yields performance 'comparable to or surpassing' full-data training is contradicted by the GSM8K row for Qwen2.5-0.5B: UFO achieves 46.27% versus 52.66% for full data, a drop of 6.39 points, which is neither comparable nor surpassing. The same row also appears without comment in the DAPO-MATH-17K block, where UFO (41.43%) actually beats full data (12.44%), so the failure is specific to the GSM8K-trained setting. The abstract, introduction, and conclusion should be revised to state the condition under which the claim holds, or the discrepancy should be explained and discussed.
  2. [Abstract, Introduction, and Section 5.3, Table 6] The manuscript claims 'less than 1/16 of the computational resources' (Abstract, Introduction, and Conclusion), but Table 6 reports speedups of 11x, 12x, 13x, 14x, 14x, and 16x. A 16x speedup corresponds to exactly 1/16 of the resources, not less than 1/16, and several entries are below 16x. The abstract's earlier phrasing 'up to a 16x reduction' is consistent with the table, but the 'less than 1/16' claim is unsupported and should be corrected or removed.
  3. [Section 4.3 and Section 3.3] The operationalization of the ZPD is the load-bearing conceptual step, and it is not justified by the preliminary study. Section 3.3 (Figure 1) establishes that data in intermediate sampling-accuracy bins improve learning more than extreme bins, but Section 4.3 defines the selected set as the top 10% by Score(s_i)=1-(s_i-mu)^2, where mu is the mean confidence of the candidate dataset. Nothing in Section 3 shows that the mean of the accuracy or confidence distribution coincides with the optimal bin. Table 1 shows strongly skewed distributions (e.g., Qwen2.5-7B has 49.07% of examples at 100% sampling accuracy; Qwen2.5-0.5B has 37.31% below 15%), so the mean can sit far from the intermediate-difficulty region shown to be optimal in Figure 1. The paper provides no sensitivity analysis varying mu and no check that the selected 10% actually falls in the accuracy bins that maximized learning in the preliminary study. Because the paper attributes its gains to the ZPD mechanism, this gap needs to be addressed with additional experiments or a clearly stated limitation; Appendix A.1 does not mention it.
  4. [Section 5.2, Table 5] The main performance comparisons are reported without variance or confidence intervals for the UFO-RL runs, even though the Random baseline is averaged over 5 runs. Many differences between UFO and Full Data are within 1-2 accuracy points (e.g., Qwen2.5-1.5B GSM8K: 76.63 vs. 76.78), which may be within run-to-run noise. Reporting multiple seeds for the main UFO condition, or at least providing error bars, is necessary to support the 'comparable or surpassing' claim. Additionally, the selection budget of 10% is a free parameter with no sensitivity analysis; a reader cannot tell whether the result depends on the particular budget or on the mean-centered score.
minor comments (6)
  1. [Table 2] The 'Similarity' metric is not defined; please specify whether it is Pearson correlation, Spearman rank correlation, or something else.
  2. [Figure 1] The x-axis is labeled 0 through 8, but the text says the data are partitioned into K=10 equally sized bins G0 through G9. Please correct the axis or the description.
  3. [Table 3] The table formatting is difficult to read because the 'Accuracy' and 'Confidence' rows are run together with the column headers; please align the values clearly.
  4. [Table 4 and Table 6] The two-column-per-model layout is visually confusing; please restructure so that each model's time and speedup are in separate labeled rows or columns.
  5. [Section 4.1] The definition Conf(x_i) uses log-probabilities, which are negative; it would help to state explicitly that s_i = exp(Conf(x_i)) is the geometric mean of token probabilities, since this is used in the filtering score.
  6. [Section 5.1 and Appendix A.5] The paper does not release code or trained-model checkpoints, and Appendix A.5 does not state the number of RL steps or the total training budget. Please include these details to improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the 10%-data claim is an empirical, held-out benchmark result; the mean-confidence ZPD anchor is an operational assumption, not a fitted input.

full rationale

UFO-RL's central claim is an empirical comparison on held-out benchmarks. Section 3.3 first establishes a non-monotonic relationship between sampling-accuracy bins and learning outcomes; Section 4.2 validates the single-pass confidence score against sampling accuracy; and Section 4.3 defines the selection score as Score(s_i)=1-(s_i-mu)^2 with mu being the mean confidence of the candidate dataset. This score is a fixed operationalization of "intermediate difficulty," not a parameter fitted to test-set performance, and no equation in the selection procedure is defined in terms of downstream accuracy. The mean-confidence anchor is an assumption, and it may be fragile for the skewed confidence distributions in Table 1, but an unsupported or questionable operationalization is a correctness and robustness concern rather than evidence that the prediction reduces by construction to its inputs. The 10% budget and the mean anchor are chosen outside the evaluation loop, and the downstream result that training on 10% near-mean-confidence data can match or exceed full-data training is contingent and falsifiable on held-out benchmarks such as GSM8K, Math500, and MMLU. The authors' self-citations in the related-work section do not carry any load-bearing premise of the derivation, and there is no fitted parameter renamed as a prediction. Therefore, no significant circularity is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on a small number of modeling choices: the 10% selection budget, the quadratic distance-from-mean score, and the assumption that mean confidence marks the ZPD. These are empirical and are only weakly motivated by the preliminary bin experiment. No code is released to check the pipeline.

free parameters (2)
  • Selection budget (10%) = 10% of candidate dataset
    The top 10% threshold is chosen without ablation; no sensitivity analysis is provided to show that 10% is optimal or robust.
  • Fuzziness score exponent = 2 (quadratic)
    The score Score(s_i)=1-(s_i-mu)^2 uses a squared distance from the mean; this functional form is asserted, not derived or ablated.
assumptions (4)
  • domain assumption ZPD theory, developed for human learners, applies to LLM reinforcement learning.
    The paper borrows the Zone of Proximal Development from educational psychology (Ref [15]) and assumes optimal learning occurs on intermediate-difficulty tasks, supported only by the preliminary bin experiment.
  • ad hoc to paper Mean confidence of the candidate dataset corresponds to the optimal difficulty for learning.
    Section 4.3 selects samples with confidence closest to mu; there is no evidence that the dataset mean, rather than some other quantile, matches the peak of the non-monotonic curve in Figure 1.
  • domain assumption Average token log-probability from a single generation is a valid proxy for multi-sample accuracy.
    Section 4.2 reports correlation (0.68-0.86) but correlation does not guarantee that the non-monotonic learning-difficulty relationship transfers from accuracy to confidence.
  • standard math Standard math and probability background (log-softmax, expected value) used in confidence definition.
    No novel mathematical machinery; standard definitions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection." pith.science (2026). https://pith.science/paper/MCRSEHQT

@misc{pith2026250512457,
  author       = {Pith},
  title        = {Pith review of: UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MCRSEHQT}},
  note         = {Machine review of arXiv:2505.12457}
}
read the original abstract

Scaling RL for LLMs is computationally expensive, largely due to multi-sampling for policy optimization and evaluation, making efficient data selection crucial. Inspired by the Zone of Proximal Development (ZPD) theory, we hypothesize LLMs learn best from data within their potential comprehension zone. Addressing the limitation of conventional, computationally intensive multi-sampling methods for data assessment, we introduce UFO-RL. This novel framework uses a computationally efficient single-pass uncertainty estimation to identify informative data instances, achieving up to 185x faster data evaluation. UFO-RL leverages this metric to select data within the estimated ZPD for training. Experiments show that training with just 10% of data selected by UFO-RL yields performance comparable to or surpassing full-data training, reducing overall training time by up to 16x while enhancing stability and generalization. UFO-RL offers a practical and highly efficient strategy for scaling RL fine-tuning of LLMs by focusing learning on valuable data.

Figures

Figures reproduced from arXiv: 2505.12457 by the authors.

Figure 1
Figure 1. Impact of Sampling Accuracy-Based Data Subset Training on Model Learning Efficiency. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 4 canonical work pages

  1. [1]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  2. [2]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/huggingface/open-r1

    2025

  3. [3]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  4. [4]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  5. [5]

    Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  6. [6]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    arXiv 2024

  7. [7]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxi...

    arXiv 2023

  8. [8]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023

    2023

Show all 28 references
  1. [9]

    Reasoning under 1 billion: Memory- augmented reinforcement learning for large language models.arXiv preprint arXiv:2504.02273, 2025

    Hung Le, Dai Do, Dung Nguyen, and Svetha Venkatesh. Reasoning under 1 billion: Memory- augmented reinforcement learning for large language models.arXiv preprint arXiv:2504.02273, 2025

    2025 arXiv

  2. [10]

    Limr: Less is more for rl scaling.arXiv preprint arXiv:2502.11886, 2025

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling.arXiv preprint arXiv:2502.11886, 2025

    2025 arXiv

  3. [11]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  4. [12]

    Wizardcoder: Empowering code large language models with evol-instruct.arXiv preprint arXiv:2306.08568, 2023

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct.arXiv preprint arXiv:2306.08568, 2023

    2023 arXiv

  5. [13]

    Tinyzero

    Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://github.com/Jiayi-Pan/TinyZero, 2025. Accessed: 2025-01-24. 10

    2025

  6. [14]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  7. [15]

    Vygotsky’s zone of proximal development: Instructional implications and teachers’ professional development.English language teaching, 3(4):237–248, 2010

    Karim Shabani, Mohamad Khatib, and Saman Ebadi. Vygotsky’s zone of proximal development: Instructional implications and teachers’ professional development.English language teaching, 3(4):237–248, 2010

    2010

  8. [16]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  9. [17]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  10. [18]

    A survey on post-training of large language models.arXiv preprint arXiv:2503.06072, 2025

    Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al. A survey on post-training of large language models.arXiv preprint arXiv:2503.06072, 2025

    2025 arXiv

  11. [19]

    Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333, 2024

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333, 2024

    2024 arXiv

  12. [20]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  13. [21]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  14. [22]

    Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387, 2025

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387, 2025

    2025 arXiv

  15. [23]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  16. [24]

    Supervised fine-tuning achieve rapid task adaption via alternating attention head activation patterns.arXiv preprint arXiv:2409.15820, 2024

    Yang Zhao, Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. Supervised fine-tuning achieve rapid task adaption via alternating attention head activation patterns.arXiv preprint arXiv:2409.15820, 2024

    2024 arXiv

  17. [25]

    Deci- phering the impact of pretraining data on large language models through machine unlearning

    Yang Zhao, Li Du, Xiao Ding, Kai Xiong, Zhouhao Sun, Shi Jun, Ting Liu, and Bing Qi. Deci- phering the impact of pretraining data on large language models through machine unlearning. InACL (Findings), 2024

    2024

  18. [26]

    Beyond similarity: A gradient-based graph method for instruction tuning data selection.arXiv preprint arXiv:2502.11062, 2025

    Yang Zhao, Li Du, Xiao Ding, Yangou Ouyang, Hepeng Wang, Kai Xiong, Jinglong Gao, Zhouhao Sun, Dongliang Xu, Yang Qing, et al. Beyond similarity: A gradient-based graph method for instruction tuning data selection.arXiv preprint arXiv:2502.11062, 2025

    2025 arXiv

  19. [27]

    Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023

    2023

  20. [28]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 11 A Technical Appendices and Supplementary Material A.1 Limi...

    1909 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.