{"id":"2461330b-bbb9-435b-89d9-264016549cbc","arxiv_id":"2505.12457","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Selecting the 10% of math problems whose model confidence is closest to the dataset average performs as well as training on all problems, at a fraction of the compute.","lead":"This paper proposes UFO-RL, a method that picks 10% of training problems for reinforcement learning by measuring how uncertain the model is about each answer. The authors report matching or beating full-data training in most cases while cutting training time by up to 16 times.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The method identifies the ZPD center with the dataset-mean confidence score, an assumption Section 4.3 introduces without evidence and that Table 5 already contradicts for Qwen2.5-0.5B.","rationale":"In good faith, the paper is trying to make RL data selection cheap by replacing multi-sample accuracy with a single-pass confidence score and then selecting the 'fuzzy middle' via closeness to the dataset mean. The confidence-accuracy correlation in Table 2 and the speedup estimates are supporting evidence, and most rows of Table 5 do show UFO-RL beating random 10% subsets and roughly matching full-data training. The load-bearing weakness is the step from 'intermediate difficulty is best' to 'intermediate equals near the dataset mean'. The preliminary study's optimal region is an accuracy decile bin, not the mean of a skewed distribution; no analysis connects the two. Because the authors provide no anchor ablation and no code, a reader cannot tell whether the selected 10% actually coincides with the ZPD or whether the gains would survive a change in the centering choice. The Table 5 Qwen2.5-0.5B GSM8K row is an additional, already-visible contradiction to the unqualified headline, but it is a qualification issue rather than the deepest structural gap. The paper is plausible and the core idea is worth testing, so I would keep the reader's conditional verdict: the central claim should not be accepted as stated until the mean-anchor assumption is tested and the headline is qualified to acknowledge the counterexample.","tokens_in":11668,"tokens_out":10254,"duration_ms":114589,"concrete_test":"Run an anchor-sensitivity experiment on one model (e.g., Qwen2.5-7B, GSM8K): fix the 10% budget and the confidence scores, then recompute the selected subset with Score centered at the median and at the 25th/75th percentiles of s_i, and also at the center of the best-performing accuracy bin from Figure 1. Report GSM8K and Math500 accuracy for each anchor. If performance is flat across anchors, the mean-centering assumption is not load-bearing; if it varies by more than 1-2 points, the central ZPD-center claim rests on an unvalidated modeling choice. As a mechanistic check, map the top-10% samples selected by the paper's rule onto the ten accuracy bins from Section 3.2 and report the overlap with the bins that produced the highest RL performance in Figure 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 defines Score(s_i)=1-(s_i-mu)^2 with mu the mean confidence of the candidate dataset and selects the top 10%. This is the entire operationalization of ZPD, but Section 3 only establishes that learning is non-monotonic when data are binned by multi-sample accuracy; it never identifies the optimal difficulty bin with the mean of the accuracy or confidence distribution. The distributions in Table 1 are strongly skewed (e.g., Qwen2.5-7B has 49.07% of training examples at 100% sampling accuracy; Qwen2.5-0.5B has 15.91% at 0% and 37.31% below 15%), so the mean can sit far from the intermediate-difficulty region that Figure 1 shows to be optimal. No experiment checks whether the 10% selected by the mean-centered score actually falls in the accuracy bins that maximized learning in the preliminary study, and no sensitivity analysis varies the anchor mu. The limitation appendix (A.1) does not mention this gap, and no code is released. This is load-bearing because the paper attributes its downstream gains to a ZPD mechanism; if the anchor is wrong, the gains may simply reflect a particular 10% subset rather than the proposed principle. Separately, Table 5 already contradicts the abstract's unqualified claim: Qwen2.5-0.5B trained on GSM8K drops from 52.66 (full data) to 46.27 (UFO).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UFO-RL, a data-selection method for reinforcement-learning fine-tuning of LLMs. The authors first present a preliminary study showing that training on data binned by multi-sample sampling accuracy has a non-monotonic effect on downstream accuracy, with intermediate-difficulty data being most beneficial. They then propose replacing expensive multi-sample accuracy with a single-pass confidence score (average log-softmax over generated tokens), define a 'fuzziness score' Score(s_i)=1-(s_i-mu)^2 centered at the mean confidence of the candidate dataset, and select the top 10% of examples by this score for RL training. Experiments on GSM8K and DAPO-MATH-17K across six models (Qwen2.5-0.5B through 7B, Llama-3.1-8B, Mistral-7B) report that this 10% subset yields performance comparable to or better than full-data training, that confidence evaluation is up to 185x faster than multi-sample accuracy evaluation, and that overall RL training time is reduced by up to 16x. The paper also claims improved training stability and generalization.","tokens_in":11984,"tokens_out":3600,"duration_ms":37067,"significance":"If the central claims hold, the paper makes a practically valuable contribution: it offers a cheap, single-pass proxy for example difficulty and shows, across several model scales and two training sets, that a small curated subset can match full-data RL fine-tuning on in-domain and near-domain benchmarks. The 185x reduction in data-evaluation cost is a concrete and useful engineering result, and the ZPD framing provides an interpretable selection principle. The empirical breadth (six models, two training datasets, three evaluation benchmarks) is a real strength. However, the significance is currently undercut by overstatements in the abstract and introduction that are contradicted by the paper's own tables, and by an under-justified operationalization of the ZPD center as the dataset-mean confidence. The paper does not release code or seeds, which limits reproducibility of the main empirical claims.","major_comments":[{"comment":"The claim that training on 10% of UFO-RL-selected data yields performance 'comparable to or surpassing' full-data training is contradicted by the GSM8K row for Qwen2.5-0.5B: UFO achieves 46.27% versus 52.66% for full data, a drop of 6.39 points, which is neither comparable nor surpassing. The same row also appears without comment in the DAPO-MATH-17K block, where UFO (41.43%) actually beats full data (12.44%), so the failure is specific to the GSM8K-trained setting. The abstract, introduction, and conclusion should be revised to state the condition under which the claim holds, or the discrepancy should be explained and discussed.","section":"Abstract and Section 5.2, Table 5"},{"comment":"The manuscript claims 'less than 1/16 of the computational resources' (Abstract, Introduction, and Conclusion), but Table 6 reports speedups of 11x, 12x, 13x, 14x, 14x, and 16x. A 16x speedup corresponds to exactly 1/16 of the resources, not less than 1/16, and several entries are below 16x. The abstract's earlier phrasing 'up to a 16x reduction' is consistent with the table, but the 'less than 1/16' claim is unsupported and should be corrected or removed.","section":"Abstract, Introduction, and Section 5.3, Table 6"},{"comment":"The operationalization of the ZPD is the load-bearing conceptual step, and it is not justified by the preliminary study. Section 3.3 (Figure 1) establishes that data in intermediate sampling-accuracy bins improve learning more than extreme bins, but Section 4.3 defines the selected set as the top 10% by Score(s_i)=1-(s_i-mu)^2, where mu is the mean confidence of the candidate dataset. Nothing in Section 3 shows that the mean of the accuracy or confidence distribution coincides with the optimal bin. Table 1 shows strongly skewed distributions (e.g., Qwen2.5-7B has 49.07% of examples at 100% sampling accuracy; Qwen2.5-0.5B has 37.31% below 15%), so the mean can sit far from the intermediate-difficulty region shown to be optimal in Figure 1. The paper provides no sensitivity analysis varying mu and no check that the selected 10% actually falls in the accuracy bins that maximized learning in the preliminary study. Because the paper attributes its gains to the ZPD mechanism, this gap needs to be addressed with additional experiments or a clearly stated limitation; Appendix A.1 does not mention it.","section":"Section 4.3 and Section 3.3"},{"comment":"The main performance comparisons are reported without variance or confidence intervals for the UFO-RL runs, even though the Random baseline is averaged over 5 runs. Many differences between UFO and Full Data are within 1-2 accuracy points (e.g., Qwen2.5-1.5B GSM8K: 76.63 vs. 76.78), which may be within run-to-run noise. Reporting multiple seeds for the main UFO condition, or at least providing error bars, is necessary to support the 'comparable or surpassing' claim. Additionally, the selection budget of 10% is a free parameter with no sensitivity analysis; a reader cannot tell whether the result depends on the particular budget or on the mean-centered score.","section":"Section 5.2, Table 5"}],"minor_comments":[{"comment":"The 'Similarity' metric is not defined; please specify whether it is Pearson correlation, Spearman rank correlation, or something else.","section":"Table 2"},{"comment":"The x-axis is labeled 0 through 8, but the text says the data are partitioned into K=10 equally sized bins G0 through G9. Please correct the axis or the description.","section":"Figure 1"},{"comment":"The table formatting is difficult to read because the 'Accuracy' and 'Confidence' rows are run together with the column headers; please align the values clearly.","section":"Table 3"},{"comment":"The two-column-per-model layout is visually confusing; please restructure so that each model's time and speedup are in separate labeled rows or columns.","section":"Table 4 and Table 6"},{"comment":"The definition Conf(x_i) uses log-probabilities, which are negative; it would help to state explicitly that s_i = exp(Conf(x_i)) is the geometric mean of token probabilities, since this is used in the filtering score.","section":"Section 4.1"},{"comment":"The paper does not release code or trained-model checkpoints, and Appendix A.5 does not state the number of RL steps or the total training budget. Please include these details to improve reproducibility.","section":"Section 5.1 and Appendix A.5"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful core idea and a broad evaluation, but the abstract and introduction overclaim on two points that are directly contradicted by Tables 5 and 6, and the central ZPD anchor (the dataset mean) is not validated. These issues are fixable with revised claims and additional experiments, so I recommend major revision rather than rejection, provided the authors address the Table 5 discrepancy and the mean-anchor sensitivity explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zhao et al. put a real efficiency idea on the table: estimate per-sample uncertainty with one forward pass (average log-softmax), select the 10% of training data whose confidence sits closest to the dataset mean, and run RL on that subset. On most of their benchmark/model combinations, that 10% subset matches or beats full-data training. That is a genuine and practically useful result, especially because the evaluation cost drops by up to two orders of magnitude relative to 16-sample accuracy scoring. The preliminary study (Figure 1) showing a non-monotonic relationship between sampling accuracy and RL gains is also a solid empirical contribution, and the comparison against high-confidence, low-confidence, and random subsets is the right experimental design.\n\nThe soft spots are real but not fatal. The abstract and conclusion claim performance 'comparable to or surpassing' full-data training without qualification, and Table 5 gives a clean counterexample: Qwen2.5-0.5B trained on GSM8K drops from 52.66 to 46.27 on the in-domain test set. 'Less than 1/16 of the computational resources' is also not supported by Table 6, whose maximum speedup is exactly 16x. The bigger conceptual gap is Section 4.3: the fuzziness score centers the ZPD on the dataset-mean confidence, but Section 3 only establishes that intermediate-difficulty bins are best, never that the mean of a skewed confidence distribution identifies that optimal bin. Given that Qwen2.5-7B has 49% of samples at 100% sampling accuracy, the mean can sit well away from the region Figure 1 flags as optimal. The authors never check whether the selected 10% actually falls in the bins that drove learning, and they do not vary the anchor or budget. No code is released, so reproducibility rests on the incomplete description.\n\nStill, the central mechanism is not circular: the selected subsets are evaluated on held-out benchmarks, and the main comparison is against clear baselines. I would send this to peer review, but the authors need to soften the claims, add error bars or repeated seeds, run a sensitivity analysis on the score form and budget, and address the Qwen2.5-0.5B regression explicitly. The paper is for people working on efficient RL fine-tuning; they will find the single-pass confidence metric and the intermediate-difficulty evidence worth engaging with.","headline":"A useful single-pass data-selection heuristic with a plausible ZPD story, but the headline claims outrun the evidence: one model clearly regresses and the mean-confidence anchor is unjustified.","tokens_in":12540,"tokens_out":2047,"would_cite":true,"duration_ms":19412,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selecting the 10% of RL training examples nearest the dataset's mean confidence matches or beats full-data training.","keywords":["reinforcement learning","data selection","uncertainty estimation","large language models","zone of proximal development","mathematical reasoning","training efficiency"],"falsifier":"On a fresh model and dataset, compute per-decile learning curves as in Figure 1, and then compare the decile containing the dataset-mean confidence with the empirically best decile; if the mean-confidence decile is not among the top bins, or if a selector centered at a different quantile beats $\\mathrm{Score}(s_i) = 1 - (s_i - \\mu)^2$, the load-bearing assumption is falsified.","tokens_in":11449,"feed_emoji":"🎯","tokens_out":7453,"duration_ms":68952,"temperature":0.7,"pith_summary":"This paper argues that the most valuable data for reinforcement-learning fine-tuning of large language models are examples of intermediate difficulty: problems the model has not mastered but can plausibly learn. It proposes a single forward-pass confidence score, the average log-probability of the generated tokens, and ranks training examples by how close their confidence is to the dataset mean. Training on the top 10% of examples by this ranking, the paper reports, matches or beats training on the full dataset on mathematical reasoning benchmarks, while cutting data-evaluation cost by up to 185x and total fine-tuning time by up to 16x. If correct, this would make large-scale RL fine-tuning substantially cheaper and more stable. The practical appeal is that the selection rule is cheap enough to run inside a training loop rather than as a separate multi-sample evaluation stage.","feed_headline":"10% of training data, chosen by one-pass uncertainty, beats full data","feed_subtitle":"A single forward pass finds 'fuzzy' mid-hard problems; 10% of them match or beat full-data RL at up to 16x lower cost.","key_machinery":"The load-bearing object is the fuzziness score $\\mathrm{Score}(s_i) = 1 - (s_i - \\mu)^2$, with $s_i = \\exp(\\mathrm{Conf}(x_i))$ the geometric mean of token probabilities from one decoding pass and $\\mu$ the mean of $s_i$ over the candidate dataset. This score operationalizes the Zone of Proximal Development as 'closest to the dataset-average confidence': samples exactly at $\\mu$ get score 1, and the top 10% by this score are the training set. The efficiency carrier is the confidence measure itself, $\\mathrm{Conf}(x_i)$, which requires a single forward pass and no iterative next-token rollouts, making data evaluation up to 185x faster than 16-sample accuracy estimation while providing a continuous, fine-grained ranking.","core_discovery":"The central discovery is that RL training value is a non-monotonic function of the model's current uncertainty: data with intermediate uncertainty produce the largest learning gains, while very easy data yield diminishing returns and very hard data destabilize training. The paper then claims that a continuous, single-pass confidence estimate—the average log-probability of output tokens, written $\\mathrm{Conf}(x_i) = \\frac{1}{T}\\sum_t \\log P(y_t \\mid x_i, y_{