Pith. sign in

REVIEW 3 major objections 6 minor 46 references

PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adaptive rollout allocation is unbiased only if it weights observed pairs of rollouts by their joint inclusion probability.

desk verdict The pairwise target-correction idea is real and the theory is honestly scoped and checks out; the empirical case is accept-shaped but needs error bars and an audit of the deployed approximation before I'd trust the +1.2/+1.4. read the letter →

arxiv 2608.11368 v2 pith:244GEGHR submitted 2026-08-11 cs.LG

Pixel Nomand , Elena Voss , Marcus Hale , Sofia Reyes This is my paper · ORCID
classification cs.LG
keywords pairwiseinclusionreweightingRLVRrolloutallocationleave-one-outgradientU-statisticsHorvitz-Thompsoncontrastgraphtokenbudget
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PAIR is built on a mismatch: the unclipped leave-one-out group-relative gradient used in GRPO-style RLVR is not a sum over individual rollouts but an average over pairs, $h_{ij}=\tfrac12(r_i-r_j)(s_i-s_j)$, so a completed rollout carries contrast only with other completed rollouts. Adaptive rollout allocation changes which pairs are observed, and averaging observed gradients as if the pair set were uniform biases the update toward the selected distribution. PAIR generates short prefixes, predicts final correctness and remaining token cost, then assigns strictly positive continuation probabilities under an expected suffix-token budget; every induced pair edge is inverse-weighted by its logged joint inclusion probability $\rho_{ij}$. Theorem 2 proves that this induced-edge estimator is design-unbiased for the complete candidate-pair gradient under the paper's scoped assumptions, and the empirical claim is that the correction transfers to compute-matched training. If correct, adaptive rollout allocation can be made unbiased with respect to the pairwise target while using about half of full-group GRPO's generated tokens: average accuracy rises by +1.2 and +1.4 points over the strongest pointwise allocator on Qwen3-1.7B and Qwen3-4B.

What carries the argument

The contrast graph is the paper's central object: prefixes are vertices, pair-gradient terms $h_{ij}$ are edges, and suffix cost is paid per vertex while statistical value lives on induced edges. PAIR constructs edge proxies $a_{ij}\approx\mathbb{E}[\|h_{ij}\|_2^2]$ and shared-vertex covariances $b_i$, then solves the convex program $\min_x \sum_{i<j}a_{ij}e^{x_i+x_j}+\sum_i b_ie^{x_i}$ subject to $\sum_i \hat c_i e^{-x_i}\le C$ and $0\le x_i\le -\log\pi_{\min}$, where $x_i=-\log\pi_i$. The resulting $\pi_i$ are strictly positive continuation probabilities, and the induced-edge estimator corrects each observed pair by its logged joint inclusion probability. The shared-vertex term is what makes vertex sampling different from independent edge sampling: completing one rollout exposes all incident edges at no extra generation cost.

What would settle it

Run the frozen-population audit with a lookahead design: log inclusion probabilities before terminal rewards are observed, then reweight with probabilities recomputed after seeing rewards, so the pre-outcome logging condition of Theorem 2 is violated. If this post-outcome design keeps PAIR's near-zero bias (1.9e-3) and its reported accuracy gains, the theorem's central assumption is not what produces the empirical result; if bias appears, the assumption is load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that the correct statistical unit of an adaptive rollout allocator is an edge of a contrast graph, not a point. For conditionally i.i.d. on-policy rollouts, the leave-one-out gradient equals the complete pair average $\hat g_{\mathrm{pair}}=\frac1{M}\sum_{i<j}h_{ij}$ (Theorem 1), so the target is a second-order U-statistic over candidate rollouts. If suffixes are selected by indicators $J_i$, an edge is observed only when $J_iJ_j=1$, and recovering the finite-population target requires both endpoints' joint inclusion probability $\rho_{ij}=\Pr(J_iJ_j=1\mid z_{1:G})$ to be logged before rewards are seen. With fresh randomization, stable potential outcomes, and $\rho_{ij}>0$, the estimator $\hat g_{\mathrm{PAIR}}=\frac1M\sum_{i<j}\frac{J_iJ_j}{\rho_{ij}}h_{ij}$ is design-unbiased (Theorem 2); the paper presents this as the principled fix for the selection bias that unweighted adaptive allocation introduces. The practical claim is that, deployed with prefix heads, a convex allocation program, and PPO-style clipping treated as an approximation, PAIR beats the strongest pointwise allocator on compute-matched runs while spending roughly half of GRPO's tokens.

Load-bearing premise

The exact design-unbiasedness guarantee holds only for the unclipped, unstandardized, on-policy leave-one-out target with probabilities logged before outcomes are seen, and the paper relies on that guarantee carrying through the practical pipeline that adds PPO clipping, reward standardization, and learned prefix predictors.

Editorial extensions

If this is right

  • Unweighted adaptive selection changes the optimand: with the complete-pair denominator kept, its conditional bias is $\frac1M\sum_{i<j}(\rho_{ij}-1)h_{ij}$, so any budget rule that ignores pair inclusion cannot target the full leave-one-out gradient.
  • Marginal per-rollout propensities are insufficient; the ablations show marginal-only weighting reaches 47.6% accuracy and MSE 0.195, while joint inclusion reweighting reaches 49.2% and 0.145.
  • On a frozen candidate population, PAIR recovers near-zero bias (1.9e-3) at matched suffix cost, confirming that the correction mechanism itself, not just the learned predictor, restores the target.
  • At compute-matched budgets PAIR surpasses full-group GRPO and the strongest pointwise allocator on both backbones: +1.2 and +1.4 average accuracy over DUET at 0.49 and 0.48 of GRPO's token budget.
  • Self-normalized weights reduce gradient MSE to 0.139 but introduce explicit bias, illustrating that unbiasedness and variance are traded off rather than jointly optimized.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The vertex/edge separation should transfer to any pairwise objective in RLVR, not just leave-one-out score gradients: any target that is a U-statistic over rollouts will need joint inclusion weights under adaptive endpoint selection.
  • If the learned edge proxies were replaced by exact pair-kernel energies, the paper's own oracle audit suggests gradient MSE could drop from 0.145 toward 0.121; this gives a testable upper bound on how much of the gain comes from predictor quality rather than the design.
  • A direct extension is to make the prefix heads themselves adaptive to the allocation: because the heads are trained under inverse-probability weighting, a policy that changes its continuation rule will shift the label distribution, and the logged probabilities provide the correction a self-training variant would need.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. PAIR addresses compute allocation in reinforcement learning with verifiable rewards (RLVR). The paper's starting point is that the unclipped leave-one-out group-relative gradient, ĝ_LOO = (1/G)Σ_i (r_i − r̄_{−i}) s_i, equals the average of pair kernels h_ij = (1/2)(r_i − r_j)(s_i − s_j) over all i<j (Theorem 1). Because adaptive continuation decides which rollout endpoints are completed, the observed pair terms are selectively revealed, and the paper argues that pointwise utility models mismatch the pairwise structure of this target. PAIR generates short prefixes, predicts correctness and remaining suffix cost with learned heads, allocates strictly positive continuation probabilities through a convex program over a pair-energy proxy (Eqs. 10–11), and estimates the complete-pair gradient with the induced-edge Horvitz–Thompson estimator ĝ_PAIR = (1/M)Σ_{i<j} (J_i J_j / ρ_ij) h_ij, logging ρ_ij before terminal rewards are observed (Eq. 12; Theorem 2; Propositions 1–2). Under an unclipped, unstandardized, on-policy target, Theorem 2 gives design-unbiasedness, which the paper explicitly scopes away from the deployed variant. Compute-matched RLVR runs on Qwen3-1.7B and 4B report 49.2% and 57.1% average accuracy, +1.2 and +1.4 over DUET, at 0.49× and 0.48× the GRPO token budget. A frozen-population audit (Table 2) shows that unweighted adaptive selection is biased (2.27e-2) while PAIR recovers near-zero bias (1.9e-3) at matched cost, with higher MSE (0.145 vs 0.050).

Significance. If the results hold, PAIR is a solid methodological contribution: it correctly identifies the pair as the statistical unit of the unclipped LOO group-relative gradient and gives a clean design-based correction for adaptive endpoint selection. I verified the core algebra by independent expansion: Theorem 1's identity is correct, Proposition 1's bias and covariance formulas match a direct calculation, and the convexity argument of Proposition 2 is sound. Strengths worth naming: the frozen-population audit compares against an externally computed complete-pair target, not a self-defined quantity; Theorem 2's scope is stated honestly and each practical departure (standardization, clipping, estimated propensities, self-normalization) is derived in Appendix B; the evaluation is pre-registered and compute-matched with decontamination controls; and the authors state an intent to release the allocation solver.

major comments (3)
  1. [§5 Theorem 2; §6 Table 1; Limitations] The deployed update in the main experiments lies outside the scope of Theorem 2 by the paper's own account: Table 4 fixes two PPO epochs with clip range 0.2 for the practical runs, Appendix B shows that reward standardization makes the update proportional to ĝ_pair/σ̂_r with E[ĝ_pair/σ̂_r] ≠ g/σ_r, and the Limitations state that 'the practical PAIR variant is an approximation to that target.' No quantitative bound, sensitivity analysis, or frozen audit of the deployed clipped/standardized estimator is provided; Table 2 audits only the unclipped, unstandardized HT estimator. As written, the +1.2/+1.4 accuracy gains over DUET cannot be traced to design-unbiasedness, because they could also be produced by the graph allocation, the prefix heads, or the selection rule. I request at least one of: (i) a frozen-policy audit of the deployed surrogate that measures its bias against the complete-pair target; (ii) reporting the LOO/Dr.GRPO exact-theorem experiments mentioned in Appendix B in the main comparison; or (iii) a reformulated mechanism claim that attributes the gains to the full PAIR system rather than to unbiasedness.
  2. [§6.2 Table 2 vs. §6.3 Table 3] The row 'Adaptive unweighted' in Table 2 (bias 2.27e-2, MSE 0.050) and the ablation 'w/o pair correction' in Table 3 (gradient MSE 0.218, accuracy 47.1) describe nominally the same estimator, namely the adaptive design with inverse pair-inclusion weights removed, yet they disagree by a factor of about four on frozen-policy MSE. Because the 2.1-point accuracy drop for this ablation is the primary evidence that pair-inclusion correction, rather than the graph allocation itself, drives the end-to-end gains, the paper must reconcile the two values or state explicitly how the configurations differ (checkpoints, candidate populations, or design details). Without that reconciliation, the mechanism test in Table 3 is not interpretable as isolating the inverse-weight correction.
  3. [§6.2 and §7] The audit numbers imply that for the biased estimator, bias² contributes only about 1% of its MSE (squared bias 5.2e-4 against MSE 0.050), while the unbiased estimator has roughly three times the MSE (0.145). The paper's claim that selection bias, rather than estimator variance, is what degrades downstream optimization is therefore a substantive causal assertion about optimization dynamics that the audit does not directly measure. The 'w/o pair correction' end-to-end ablation is the right kind of evidence, but its current interpretation depends on resolving the discrepancy raised in the previous comment. I ask the authors to quantify the support for this causal claim, for example by reporting the bias of the deployed surrogate alongside downstream accuracy, or by stating explicitly how the audit's bias and MSE values are meant to be traded off in training.
minor comments (6)
  1. [§6.3; Table 4; Appendix C] The recommended operating point in §6.3 sets π_min=0.05, whereas Table 4 and Appendix C give the default probability floor as π_min=0.08; please reconcile the reported default.
  2. [§D.3; Table 1] The protocol commits to seed means, standard deviations, and paired bootstrap intervals, but Table 1 reports single accuracy values without any variance information; please report the seed statistics, even if only in the appendix.
  3. [§4.3] The edge proxy bd_ij is described as the 'parameter-free default,' but it is a function of the learned prefix-head predictions b^p_i and b^p_j; if the intended meaning is 'no parameters beyond the two heads,' the text should say so.
  4. [Table 3, 'w/o suffix cost' row] Removing the suffix-cost term changes both the objective and the realized token budget (0.58 vs 0.49 of GRPO tokens), so the 48.4 accuracy figure is not a compute-matched comparison; please clarify whether this row is meant to illustrate a cost-efficiency trade-off rather than an accuracy comparison.
  5. [Abstract and §6.1] The efficiency claim of '51% and 52% fewer generated tokens than full-group GRPO' is accurate, but relative to the strongest pointwise baseline (DUET) the token reduction is only about 0.03 of the GRPO budget on both backbones; stating both comparisons would give a more precise picture of the marginal contribution over prior allocators.
  6. [§4.5, Eq. (13)] The 1/(2M) factor in the definition of eÂ_i is consistent with ĝ_PAIR because each unordered pair is counted twice when summing over vertices, but the cancellation deserves a one-line explanation to prevent readers from reading it as a discrepancy with Eq. (12).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the design-unbiasedness theorem is a standard Horvitz–Thompson identity that is independent of the fitted predictors, and the empirical gains are measured against external benchmarks.

full rationale

The paper's derivation chain is self-contained and does not reduce to its own inputs. Theorem 1 (pairwise representation) is proven in Appendix A.2 by elementary algebra, with the U-statistic representation explicitly cited to Zhou et al. (2026) but not relied on as an unverified premise. Theorem 2 (design-unbiased induced-edge estimator) is a term-by-term Horvitz–Thompson argument: with rho_ij = P(J_i J_j = 1 | z_1:G) > 0 and pre-outcome logged probabilities, E[I_ij/rho_ij | F] = 1 by definition of rho_ij, so E[ghat_PAIR | F] = ghat_pair. This guarantee is parameter-free with respect to the fitted prefix heads (Eq. 5) and the convex allocation (Eqs. 10-11); those components affect variance and cost, not the unbiasedness claim. The empirical claims are externally anchored: average accuracy is reported on MATH500, AIME24, AMC23, OlympiadBench, and LiveCodeBench against compute-matched baselines, and the frozen-population audit compares against the full candidate-pool pair gradient, not a self-defined quantity. The paper also explicitly scopes the exact theorem to the unclipped, unstandardized LOO target and states in the Limitations and Appendix B that the practical PAIR variant with PPO clipping, reward standardization, and multiple epochs is 'an approximation to that target.' That is an honest correctness/scope limitation, not a circular reduction. There is no load-bearing self-citation chain and no fitted parameter that is renamed as a prediction. The mild possibility that the allocation's efficiency gains are partly due to the learned heads is an empirical confounding concern, not a definitional circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The estimator's unbiasedness rests only on standard score-function identities plus the procedural assumptions (pre-outcome logging, positivity, stable potential outcomes), all stated. The fitted heads and edge proxies influence efficiency and budget feasibility but not the central unbiasedness claim; the domination condition that would make the variance bound rigorous is not verified in practice.

free parameters (4)
  • Prefix success head bhat_p_i (MLP weights) = trained on completed rollouts via IPW Brier loss (Eq. 6)
    Predicts P(r_i=1 | prefix) from prefix hidden state (Eq. 5); feeds the edge proxy in Eq. (8). Affects allocation efficiency, not the unbiasedness theorem.
  • Prefix cost head bhat_c_i (MLP weights) = trained via IPW squared log-error (Eq. 7)
    Predicts remaining suffix tokens; appears in the budget constraint (Eq. 11). Predicted cost can deviate from realized cost, which the paper reports separately.
  • Edge proxy a_ij = default bhat_d_ij; optional bhat_d_ij(bhat_m_i+bhat_m_j)^2/4 or regressed pair-energy head
    Eq. (9) proxy for E[||h_ij||^2 | z_i,z_j]; the variance upper bound in Proposition 2 requires a_ij to dominate ||h_ij||^2, which learned proxies do not guarantee.
  • Design hyperparameters: pi_min, tau, G, budget ratio = pi_min 0.08 (Table 4) or 0.05 (Section 6.3); tau=256; G=16; budget ratio 0.5
    Hand-chosen and sweep-tuned (Table 6); the reported pi_min values are internally inconsistent between sections.
assumptions (5)
  • standard math Score-function identities (Eq. 14-15)
    Needed for E[s_i | q] = 0 and grad_theta E[r|q] = E[r_i s_i | q]; assumes differentiation passes through the expectation and reward has no direct theta derivative.
  • domain assumption Conditionally independent on-policy rollouts
    Theorem 1 and unconditional unbiasedness require o_1:G i.i.d. from pi_theta(.|q); Appendix F notes correlated rollouts (tree expansion, shared prefixes) break this.
  • domain assumption Stable potential outcomes
    Theorem 2 requires h_ij unchanged by which other vertices are continued; holds for independent prefixes with separate decoding randomness, not for coupled decoding.
  • domain assumption Fresh randomization and pre-outcome logging
    Design-unbiasedness requires rho_ij logged before terminal rewards are observed (Section 4.1, Theorem 2); deterministic outcome pruning is excluded.
  • ad hoc to paper Domination condition a_ij >= ||h_ij||^2
    Needed for the variance upper bound in Proposition 2/A.6; not guaranteed for the default disagreement proxy or learned edge proxies, so the paper demotes the bound to a surrogate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR." pith.science (2026). https://pith.science/paper/244GEGHR

@misc{pith2026260811368,
  author       = {Pith},
  title        = {Pith review of: PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/244GEGHR}},
  note         = {Machine review of arXiv:2608.11368}
}
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) spends most of its compute generating groups of long reasoning trajectories. Recent allocators reduce this cost by assigning budgets to prompts, rollouts, or tokens according to a pointwise notion of difficulty or utility. We identify a statistical mismatch: the unclipped leave-one-out group-relative score gradient is not a sum of independent point contributions, but a second-order U-statistic over pairs of rollouts. Completing one rollout therefore reveals contrast with every other completed rollout, and adaptive endpoint selection changes which pair terms are observable. We introduce PAIR (Pairwise-Aware Inclusion Reweighting), which treats short rollout prefixes as vertices and pair-gradient terms as edges of a contrast graph. A prefix-only predictor estimates correctness and remaining token cost; a convex design chooses positive continuation probabilities under an expected suffix-token budget; and each edge induced by completed vertices is inverse-weighted by its logged joint inclusion probability. Under conditionally independent on-policy rollouts and an unclipped, unstandardized objective, the resulting estimator is design-unbiased for the complete candidate-pair gradient. Across compute-matched RLVR runs on Qwen3-1.7B/4B, PAIR improves average accuracy by +1.2 and +1.4 over the strongest pointwise allocator while using 51% and 52% fewer generated tokens than full-group GRPO. A frozen-population estimator audit confirms that unweighted adaptive selection is biased, whereas pair-inclusion correction recovers the complete-pair target at matched suffix cost.

Figures

Figures reproduced from arXiv: 2608.11368 by the authors.

Figure 1
Figure 1. PAIR allocates rollout endpoints for a pairwise estimator. A short prefix probe estimates correctness and remaining generation cost before terminal rewards are observed. The contrast-graph program assigns positive continuation probabilities under a suffix-token budget. Completing a vertex exposes every incident edge to other completed vertices, and the update inverse-weights each observed edge by its logged joint in… view at source ↗
Figure 2
Figure 2. Mechanism of pair-aware allocation. (a) Binary reward contrast depends jointly on two prefix success probabilities, not on either endpoint alone. (b) PAIR spends suffix cost on vertices; continuing one vertex exposes all incident edges to other completed vertices. Opacity shows optimized continuation probability and edge width shows the pair-energy proxy. (c) On the displayed contrast graph, the convex PAIR design r… view at source ↗
Figure 3
Figure 3. Accuracy–cost trade-off and mechanism ablations. (a) Across both backbones, PAIR (stars) sits on the Pareto frontier of mean accuracy versus relative generated tokens. (b) Removing pair-inclusion correction or replacing the graph design with pointwise/uniform rules increases frozen-policy gradient MSE. Uniform HT Pointwise HT Adaptive unweighted PAIR HT 0.000 0.005 0.010 0.015 0.020 bias norm a Adaptive selection ne… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Controlled estimator audit on a frozen candidate population. Adaptive unweighted selection changes the target; inverse pair-inclusion weighting recovers the complete-pair gradient in expectation. MSE remains nonzero because unbiasedness does not remove design variance.…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 17 canonical work pages

  1. [1]

    arXiv preprint arXiv:2402.03300 , year=

    DeepSeekMath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    arXiv

  2. [2]

    arXiv preprint arXiv:2501.12948 , year=

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  3. [3]

    arXiv preprint arXiv:2503.14476 , year=

    DAPO: An open-source LLM reinforcement learning system at scale , author=. arXiv preprint arXiv:2503.14476 , year=

    arXiv

  4. [4]

    arXiv preprint arXiv:2501.03262 , year=

    REINFORCE++: Stabilizing critic-free policy optimization with global advantage normalization , author=. arXiv preprint arXiv:2501.03262 , year=

    arXiv

  5. [5]

    arXiv preprint arXiv:2503.20783 , year=

    Understanding R1-Zero-like training: A critical perspective , author=. arXiv preprint arXiv:2503.20783 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2504.03380 , year=

    Online difficulty filtering for reasoning oriented reinforcement learning , author=. arXiv preprint arXiv:2504.03380 , year=

    arXiv

  7. [7]

    arXiv preprint arXiv:2505.14970 , year=

    Self-evolving curriculum for LLM reasoning , author=. arXiv preprint arXiv:2505.14970 , year=

    arXiv

  8. [8]

    arXiv preprint arXiv:2506.02177 , year=

    Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts , author=. arXiv preprint arXiv:2506.02177 , year=

    arXiv

Show all 46 references
  1. [9]

    arXiv preprint arXiv:2509.25808 , year=

    Improving sampling efficiency in RLVR through adaptive rollout and response reuse , author=. arXiv preprint arXiv:2509.25808 , year=

  2. [10]

    arXiv preprint arXiv:2508.13755 , year=

    Depth-breadth synergy in RLVR: Unlocking LLM reasoning gains with adaptive exploration , author=. arXiv preprint arXiv:2508.13755 , year=

    arXiv

  3. [11]

    International Conference on Learning Representations , volume=

    Dynamics-predictive sampling for active RL finetuning of large reasoning models , author=. International Conference on Learning Representations , volume=

  4. [12]

    arXiv preprint arXiv:2602.01970 , year=

    Small generalizable prompt predictive models can steer efficient rl post-training of large reasoning models , author=. arXiv preprint arXiv:2602.01970 , year=

    arXiv

  5. [13]

    arXiv preprint arXiv:2602.01601 , year=

    Adaptive rollout allocation for online reinforcement learning with verifiable rewards , author=. arXiv preprint arXiv:2602.01601 , year=

  6. [14]

    arXiv preprint arXiv:2605.07114 , year=

    Where to spend rollouts: Hit-utility optimal rollout allocation for group-based RLVR , author=. arXiv preprint arXiv:2605.07114 , year=

    arXiv

  7. [15]

    arXiv preprint arXiv:2607.22002 , year=

    Learning as reasoning unfolds: Progressive rollout allocation for efficient reinforcement learning , author=. arXiv preprint arXiv:2607.22002 , year=

    arXiv

  8. [16]

    arXiv preprint arXiv:2606.11119 , year=

    TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning , author=. arXiv preprint arXiv:2606.11119 , year=

    arXiv

  9. [17]

    arXiv preprint arXiv:2603.04135 , year=

    Unbiased dynamic pruning for efficient group-based policy optimization , author=. arXiv preprint arXiv:2603.04135 , year=

  10. [18]

    arXiv preprint arXiv:2605.08441 , year=

    DUET: Optimize token-budget allocation for reinforcement learning with verifiable rewards , author=. arXiv preprint arXiv:2605.08441 , year=

    arXiv

  11. [19]

    arXiv preprint arXiv:2606.05606 , year=

    Cross-epoch adaptive rollout optimization for RL post-training , author=. arXiv preprint arXiv:2606.05606 , year=

    arXiv

  12. [20]

    arXiv preprint arXiv:2606.08854 , year=

    sGPO: Trading inference FLOPs for training efficiency in RLVR , author=. arXiv preprint arXiv:2606.08854 , year=

    arXiv

  13. [21]

    arXiv preprint arXiv:2606.17250 , year=

    Rethinking groups in critic-free RLVR , author=. arXiv preprint arXiv:2606.17250 , year=

    arXiv

  14. [22]

    arXiv preprint arXiv:2603.01162 , year=

    Demystifying group relative policy optimization: Its policy gradient is a U-statistic , author=. arXiv preprint arXiv:2603.01162 , year=

  15. [23]

    arXiv preprint arXiv:2603.06619 , year=

    Not all tokens are needed: Token-efficient reinforcement learning , author=. arXiv preprint arXiv:2603.06619 , year=

  16. [24]

    arXiv preprint arXiv:2606.02345 , year=

    Doing well with less! On sampling techniques for empirical pairwise loss estimation/minimization , author=. arXiv preprint arXiv:2606.02345 , year=

    arXiv

  17. [25]

    arXiv preprint arXiv:2601.15609 , year=

    When sharpening becomes collapse: Sampling bias and semantic coupling in reinforcement learning with verifiable rewards , author=. arXiv preprint arXiv:2601.15609 , year=

  18. [26]

    arXiv preprint arXiv:2602.21492 , year=

    GradAlign: Gradient-aligned data selection for LLM reinforcement learning , author=. arXiv preprint arXiv:2602.21492 , year=

    arXiv

  19. [27]

    arXiv preprint arXiv:2510.00819 , year=

    Stabilizing policy gradients for sample-efficient reinforcement learning in LLM reasoning , author=. arXiv preprint arXiv:2510.00819 , year=

  20. [28]

    Journal of the American Statistical Association , volume=

    A generalization of sampling without replacement from a finite universe , author=. Journal of the American Statistical Association , volume=

  21. [29]

    The Annals of Mathematical Statistics , volume=

    A class of statistics with asymptotically normal distribution , author=. The Annals of Mathematical Statistics , volume=

  22. [30]

    Journal of the Royal Statistical Society , volume=

    On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection , author=. Journal of the Royal Statistical Society , volume=

  23. [31]

    The Annals of Mathematical Statistics , volume=

    Asymptotic theory of rejective sampling with varying probabilities from a finite population , author=. The Annals of Mathematical Statistics , volume=

  24. [32]

    Model assisted survey sampling , author=

  25. [33]

    Statistica Sinica , volume=

    Design based incomplete U-statistics , author=. Statistica Sinica , volume=

  26. [34]

    Proceedings of the 31st Conference on Learning Theory , volume=

    Counting motifs with graph sampling , author=. Proceedings of the 31st Conference on Learning Theory , volume=

  27. [35]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    arXiv

  28. [36]

    arXiv preprint arXiv:2402.14740 , year=

    Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs , author=. arXiv preprint arXiv:2402.14740 , year=

    arXiv

  29. [37]

    arXiv preprint arXiv:2103.03874 , year=

    Measuring mathematical problem solving with the MATH dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    arXiv

  30. [38]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  31. [39]

    arXiv preprint arXiv:2305.20050 , year=

    Let's verify step by step , author=. arXiv preprint arXiv:2305.20050 , year=

    arXiv

  32. [40]

    arXiv preprint arXiv:2312.14852 , year=

    TACO: Topics in algorithmic code generation dataset , author=. arXiv preprint arXiv:2312.14852 , year=

    arXiv

  33. [41]

    arXiv preprint arXiv:2402.14008 , year=

    OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems , author=. arXiv preprint arXiv:2402.14008 , year=

    arXiv

  34. [42]

    arXiv preprint arXiv:2403.07974 , year=

    LiveCodeBench: Holistic and contamination free evaluation of large language models for code , author=. arXiv preprint arXiv:2403.07974 , year=

    arXiv

  35. [43]

    arXiv preprint arXiv:2311.12022 , year=

    GPQA: A graduate-level google-proof Q&A benchmark , author=. arXiv preprint arXiv:2311.12022 , year=

    arXiv

  36. [44]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  37. [45]

    Proceedings of the Twentieth European Conference on Computer Systems , year=

    HybridFlow: A flexible and efficient RLHF framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , year=

  38. [46]

    Proceedings of the 29th ACM Symposium on Operating Systems Principles , pages=

    Efficient memory management for large language model serving with PagedAttention , author=. Proceedings of the 29th ACM Symposium on Operating Systems Principles , pages=

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.