{"id":"cd401752-3ea9-4d74-9292-48058a97f4bc","arxiv_id":"2608.11368","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adaptive rollout allocation in RLVR is biased if it ignores pairwise contrasts; PAIR adds logged joint-inclusion reweighting so the estimate stays unbiased for the complete candidate-pair gradient.","lead":"PAIR is a new training-efficiency method for reasoning models that treats the learning signal as pairs of rollouts rather than individual rollouts, and corrects for which pairs an adaptive budget rule happens to observe. It reports similar or better reasoning accuracy than prior allocators while generating only about half the tokens of full-group GRPO.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Practical PAIR update is outside Theorem 2's guarantee, with no quantitative bound on the approximation; empirical gains cannot be attributed to unbiasedness without that audit.","rationale":"Theorems 1 and 2 are correctly argued: the pairwise representation identity checks out, and the Horvitz-Thompson argument is valid under the stated procedural assumptions. The paper is also admirably honest in scoping the guarantee. But honesty about the scope does not bridge the gap: the headline experimental numbers are produced by a clipped, standardized, multi-epoch variant that is not the estimator whose unbiasedness is proved. Without a quantitative measure of the approximation error, the theoretical result and the empirical result are decoupled. The frozen audit (Table 2) cannot fill this gap because it evaluates the exact HT estimator, not the deployed update. A second, reinforcing issue is the small reported gains without seed-level error bars; the claimed significance criterion (paired bootstrap excludes zero) is described but not shown. The pi_min inconsistency (0.05 in Section 6 vs 0.08 in Appendix C and Table 4) is minor and does not affect the theorems. Overall, the correct verdict is conditional: the central mathematical claim appears sound, but the empirical validation does not yet demonstrate that the deployed method inherits it. A focused audit of the deployed update against the exact target would settle the matter.","tokens_in":17923,"tokens_out":13617,"duration_ms":124419,"concrete_test":"On each frozen checkpoint from Section 6.2, compute the exact complete-pair target ghat_pair, then compute the actual PAIR update as deployed in Table 4 (two PPO epochs, clip range 0.2, group reward standardization, detached pair advantages, logged probabilities) over the same 50,000 design draws, and report the bias norm and MSE of the deployed update against ghat_pair. If the bias is of order 1e-2 (comparable to the 2.27e-2 adaptive-unweighted bias in Table 2), the practical estimator is not the unbiased estimator Theorem 2 covers, and the empirical gains must be attributed to another mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim (Theorem 2, Section 5) guarantees design-unbiasedness of ghat_PAIR for the unclipped, unstandardized leave-one-out pair target. The deployed update in the main experiments (Section 6, Table 1) is, by the paper's own Limitations and Appendix B, a different estimator: it applies group reward standardization, PPO clipping, and multiple epochs. Appendix B notes that standardization makes the update proportional to ghat_pair / sigma_r, whose expectation is not the normalized target, and that clipping replaces the gradient with a trust-region surrogate. The paper provides no quantitative bound or frozen audit of the distance between this deployed update and the exact target; the audit in Table 2 measures only the unclipped, unstandardized HT estimator. As a result, the reported +1.2 and +1.4 accuracy gains cannot be traced to design-unbiasedness; they could be produced by the biased selection, the graph allocation, or the prefix heads. The paper honestly concedes that 'the practical PAIR variant is an approximation to that target,' but a concession without a measurement leaves the central claim untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"PAIR addresses compute allocation in reinforcement learning with verifiable rewards (RLVR). The paper's starting point is that the unclipped leave-one-out group-relative gradient, ĝ_LOO = (1/G)Σ_i (r_i − r̄_{−i}) s_i, equals the average of pair kernels h_ij = (1/2)(r_i − r_j)(s_i − s_j) over all i<j (Theorem 1). Because adaptive continuation decides which rollout endpoints are completed, the observed pair terms are selectively revealed, and the paper argues that pointwise utility models mismatch the pairwise structure of this target. PAIR generates short prefixes, predicts correctness and remaining suffix cost with learned heads, allocates strictly positive continuation probabilities through a convex program over a pair-energy proxy (Eqs. 10–11), and estimates the complete-pair gradient with the induced-edge Horvitz–Thompson estimator ĝ_PAIR = (1/M)Σ_{i<j} (J_i J_j / ρ_ij) h_ij, logging ρ_ij before terminal rewards are observed (Eq. 12; Theorem 2; Propositions 1–2). Under an unclipped, unstandardized, on-policy target, Theorem 2 gives design-unbiasedness, which the paper explicitly scopes away from the deployed variant. Compute-matched RLVR runs on Qwen3-1.7B and 4B report 49.2% and 57.1% average accuracy, +1.2 and +1.4 over DUET, at 0.49× and 0.48× the GRPO token budget. A frozen-population audit (Table 2) shows that unweighted adaptive selection is biased (2.27e-2) while PAIR recovers near-zero bias (1.9e-3) at matched cost, with higher MSE (0.145 vs 0.050).","tokens_in":18008,"tokens_out":25691,"duration_ms":207698,"significance":"If the results hold, PAIR is a solid methodological contribution: it correctly identifies the pair as the statistical unit of the unclipped LOO group-relative gradient and gives a clean design-based correction for adaptive endpoint selection. I verified the core algebra by independent expansion: Theorem 1's identity is correct, Proposition 1's bias and covariance formulas match a direct calculation, and the convexity argument of Proposition 2 is sound. Strengths worth naming: the frozen-population audit compares against an externally computed complete-pair target, not a self-defined quantity; Theorem 2's scope is stated honestly and each practical departure (standardization, clipping, estimated propensities, self-normalization) is derived in Appendix B; the evaluation is pre-registered and compute-matched with decontamination controls; and the authors state an intent to release the allocation solver.","major_comments":[{"comment":"The deployed update in the main experiments lies outside the scope of Theorem 2 by the paper's own account: Table 4 fixes two PPO epochs with clip range 0.2 for the practical runs, Appendix B shows that reward standardization makes the update proportional to ĝ_pair/σ̂_r with E[ĝ_pair/σ̂_r] ≠ g/σ_r, and the Limitations state that 'the practical PAIR variant is an approximation to that target.' No quantitative bound, sensitivity analysis, or frozen audit of the deployed clipped/standardized estimator is provided; Table 2 audits only the unclipped, unstandardized HT estimator. As written, the +1.2/+1.4 accuracy gains over DUET cannot be traced to design-unbiasedness, because they could also be produced by the graph allocation, the prefix heads, or the selection rule. I request at least one of: (i) a frozen-policy audit of the deployed surrogate that measures its bias against the complete-pair target; (ii) reporting the LOO/Dr.GRPO exact-theorem experiments mentioned in Appendix B in the main comparison; or (iii) a reformulated mechanism claim that attributes the gains to the full PAIR system rather than to unbiasedness.","section":"§5 Theorem 2; §6 Table 1; Limitations"},{"comment":"The row 'Adaptive unweighted' in Table 2 (bias 2.27e-2, MSE 0.050) and the ablation 'w/o pair correction' in Table 3 (gradient MSE 0.218, accuracy 47.1) describe nominally the same estimator, namely the adaptive design with inverse pair-inclusion weights removed, yet they disagree by a factor of about four on frozen-policy MSE. Because the 2.1-point accuracy drop for this ablation is the primary evidence that pair-inclusion correction, rather than the graph allocation itself, drives the end-to-end gains, the paper must reconcile the two values or state explicitly how the configurations differ (checkpoints, candidate populations, or design details). Without that reconciliation, the mechanism test in Table 3 is not interpretable as isolating the inverse-weight correction.","section":"§6.2 Table 2 vs. §6.3 Table 3"},{"comment":"The audit numbers imply that for the biased estimator, bias² contributes only about 1% of its MSE (squared bias 5.2e-4 against MSE 0.050), while the unbiased estimator has roughly three times the MSE (0.145). The paper's claim that selection bias, rather than estimator variance, is what degrades downstream optimization is therefore a substantive causal assertion about optimization dynamics that the audit does not directly measure. The 'w/o pair correction' end-to-end ablation is the right kind of evidence, but its current interpretation depends on resolving the discrepancy raised in the previous comment. I ask the authors to quantify the support for this causal claim, for example by reporting the bias of the deployed surrogate alongside downstream accuracy, or by stating explicitly how the audit's bias and MSE values are meant to be traded off in training.","section":"§6.2 and §7"}],"minor_comments":[{"comment":"The recommended operating point in §6.3 sets π_min=0.05, whereas Table 4 and Appendix C give the default probability floor as π_min=0.08; please reconcile the reported default.","section":"§6.3; Table 4; Appendix C"},{"comment":"The protocol commits to seed means, standard deviations, and paired bootstrap intervals, but Table 1 reports single accuracy values without any variance information; please report the seed statistics, even if only in the appendix.","section":"§D.3; Table 1"},{"comment":"The edge proxy bd_ij is described as the 'parameter-free default,' but it is a function of the learned prefix-head predictions b^p_i and b^p_j; if the intended meaning is 'no parameters beyond the two heads,' the text should say so.","section":"§4.3"},{"comment":"Removing the suffix-cost term changes both the objective and the realized token budget (0.58 vs 0.49 of GRPO tokens), so the 48.4 accuracy figure is not a compute-matched comparison; please clarify whether this row is meant to illustrate a cost-efficiency trade-off rather than an accuracy comparison.","section":"Table 3, 'w/o suffix cost' row"},{"comment":"The efficiency claim of '51% and 52% fewer generated tokens than full-group GRPO' is accurate, but relative to the strongest pointwise baseline (DUET) the token reduction is only about 0.03 of the GRPO budget on both backbones; stating both comparisons would give a more precise picture of the marginal contribution over prior allocators.","section":"Abstract and §6.1"},{"comment":"The 1/(2M) factor in the definition of eÂ_i is consistent with ĝ_PAIR because each unordered pair is counted twice when summing over vertices, but the cancellation deserves a one-line explanation to prevent readers from reading it as a discrepancy with Eq. (12).","section":"§4.5, Eq. (13)"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its scoped claims, and the authors' honesty about limitations is a genuine strength; my recommendation of major revision rests on two fixable gaps: the absence of an audit or quantitative bound for the actually deployed clipped/standardized estimator, and the unexplained MSE discrepancy between Tables 2 and 3 for the ablated estimator that carries the mechanism argument. I would also flag for the editor that the headline efficiency gains are relative to full-group GRPO, while the marginal improvement over the strongest pointwise baseline is small in both tokens and accuracy; the framing is accurate but the incremental contribution over DUET is modest. If the authors supply the deployed-surrogate audit and reconcile the two tables, I would expect the revised paper to be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead PAIR last night. The thing you should know: this is a genuinely good idea with an honest theoretical core, and the empirical section is the weakest part. If I were advising an editor, I'd send it out — it's a serious paper — but I'd tell the authors to tighten the reporting rather than the math.\n\nWhat's actually new: the observation that adaptive endpoint selection biases which pair terms you can observe in a pairwise LOO gradient, and that the right correction uses logged joint inclusion probabilities rather than pointwise propensities. The U-statistic representation itself is Zhou et al.'s, and HT weighting is textbook — the paper says so. The vertex-cost/edge-value framing and the graph-coupled allocation program are the original pieces, and they're clearly worked out. Theorem 1 checks out (pair average equals LOO exactly) and Theorem 2 is a correct HT argument within its scoped conditions.\n\nCredit where earned: the Limitations section is unusually honest. The paper states plainly that the practical PAIR variant is an approximation to the exact target, and Appendix B derives where clipping, standardization, and estimated propensities break the guarantee. That is the right way to scope a theory claim.\n\nSoft spots, in order of real weight. First, the frozen audit shows PAIR's unbiased estimator has higher MSE (0.145) than biased adaptive unweighted selection (0.050). The paper argues bias, not MSE, is what degrades optimization — a causal claim the experiments support only indirectly, and it's doing a lot of work in explaining the +1.2/+1.4 gains. Second, the deployed update sits outside Theorem 2's guarantee, with no quantitative bound or audit of that gap. The stress-tester's headline is fair in the narrow sense that the empirical gains can't be traced to unbiasedness; the paper doesn't overclaim exactly, but a measurement of the approximation would close it. Third, reporting: no error bars or seed-level tables in Table 1, no GSM8K/GPQA numbers in the provided text despite being claimed in the setup, and no code release. Fourth, minor: the main text recommends pi_min=0.05 while Appendix C and Table 4 say 0.08 — presumably a stale default, but it should be reconciled.\n\nFor whom: anyone working on RLVR token efficiency or adaptively sampled policy gradients. It deserves a serious referee. I'd want the revisions above before accepting, but the theory core and the problem framing are solid.","headline":"The pairwise target-correction idea is real and the theory is honestly scoped and checks out; the empirical case is accept-shaped but needs error bars and an audit of the deployed approximation before I'd trust the +1.2/+1.4.","tokens_in":18693,"tokens_out":2927,"would_cite":true,"duration_ms":24507,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adaptive rollout allocation is unbiased only if it weights observed pairs of rollouts by their joint inclusion probability.","keywords":["pairwise inclusion reweighting","RLVR","rollout allocation","leave-one-out gradient","U-statistics","Horvitz-Thompson","contrast graph","token budget"],"falsifier":"Run the frozen-population audit with a lookahead design: log inclusion probabilities before terminal rewards are observed, then reweight with probabilities recomputed after seeing rewards, so the pre-outcome logging condition of Theorem 2 is violated. If this post-outcome design keeps PAIR's near-zero bias (1.9e-3) and its reported accuracy gains, the theorem's central assumption is not what produces the empirical result; if bias appears, the assumption is load-bearing.","tokens_in":17559,"feed_emoji":"⚖️","tokens_out":8931,"duration_ms":75964,"temperature":0.7,"pith_summary":"PAIR is built on a mismatch: the unclipped leave-one-out group-relative gradient used in GRPO-style RLVR is not a sum over individual rollouts but an average over pairs, $h_{ij}=\\tfrac12(r_i-r_j)(s_i-s_j)$, so a completed rollout carries contrast only with other completed rollouts. Adaptive rollout allocation changes which pairs are observed, and averaging observed gradients as if the pair set were uniform biases the update toward the selected distribution. PAIR generates short prefixes, predicts final correctness and remaining token cost, then assigns strictly positive continuation probabilities under an expected suffix-token budget; every induced pair edge is inverse-weighted by its logged joint inclusion probability $\\rho_{ij}$. Theorem 2 proves that this induced-edge estimator is design-unbiased for the complete candidate-pair gradient under the paper's scoped assumptions, and the empirical claim is that the correction transfers to compute-matched training. If correct, adaptive rollout allocation can be made unbiased with respect to the pairwise target while using about half of full-group GRPO's generated tokens: average accuracy rises by +1.2 and +1.4 points over the strongest pointwise allocator on Qwen3-1.7B and Qwen3-4B.","feed_headline":"Pairwise reweighting cuts RLVR rollout tokens roughly in half","feed_subtitle":"PAIR shows group-relative gradients are pairwise, and weighting observed pairs by joint inclusion keeps them unbiased.","key_machinery":"The contrast graph is the paper's central object: prefixes are vertices, pair-gradient terms $h_{ij}$ are edges, and suffix cost is paid per vertex while statistical value lives on induced edges. PAIR constructs edge proxies $a_{ij}\\approx\\mathbb{E}[\\|h_{ij}\\|_2^2]$ and shared-vertex covariances $b_i$, then solves the convex program $\\min_x \\sum_{i<j}a_{ij}e^{x_i+x_j}+\\sum_i b_ie^{x_i}$ subject to $\\sum_i \\hat c_i e^{-x_i}\\le C$ and $0\\le x_i\\le -\\log\\pi_{\\min}$, where $x_i=-\\log\\pi_i$. The resulting $\\pi_i$ are strictly positive continuation probabilities, and the induced-edge estimator corrects each observed pair by its logged joint inclusion probability. The shared-vertex term is what makes vertex sampling different from independent edge sampling: completing one rollout exposes all incident edges at no extra generation cost.","core_discovery":"The central claim is that the correct statistical unit of an adaptive rollout allocator is an edge of a contrast graph, not a point. For conditionally i.i.d. on-policy rollouts, the leave-one-out gradient equals the complete pair average $\\hat g_{\\mathrm{pair}}=\\frac1{M}\\sum_{i<j}h_{ij}$ (Theorem 1), so the target is a second-order U-statistic over candidate rollouts. If suffixes are selected by indicators $J_i$, an edge is observed only when $J_iJ_j=1$, and recovering the finite-population target requires both endpoints' joint inclusion probability $\\rho_{ij}=\\Pr(J_iJ_j=1\\mid z_{1:G})$ to be logged before rewards are seen. With fresh randomization, stable potential outcomes, and $\\rho_{ij}>0$, the estimator $\\hat g_{\\mathrm{PAIR}}=\\frac1M\\sum_{i<j}\\frac{J_iJ_j}{\\rho_{ij}}h_{ij}$ is design-unbiased (Theorem 2); the paper presents this as the principled fix for the selection bias that unweighted adaptive allocation introduces. The practical claim is that, deployed with prefix heads, a convex allocation program, and PPO-style clipping treated as an approximation, PAIR beats the strongest pointwise allocator on compute-matched runs while spending roughly half of GRPO's tokens.","pith_inferences":["The vertex/edge separation should transfer to any pairwise objective in RLVR, not just leave-one-out score gradients: any target that is a U-statistic over rollouts will need joint inclusion weights under adaptive endpoint selection.","If the learned edge proxies were replaced by exact pair-kernel energies, the paper's own oracle audit suggests gradient MSE could drop from 0.145 toward 0.121; this gives a testable upper bound on how much of the gain comes from predictor quality rather than the design.","A direct extension is to make the prefix heads themselves adaptive to the allocation: because the heads are trained under inverse-probability weighting, a policy that changes its continuation rule will shift the label distribution, and the logged probabilities provide the correction a self-training variant would need."],"forward_implications":["Unweighted adaptive selection changes the optimand: with the complete-pair denominator kept, its conditional bias is $\\frac1M\\sum_{i<j}(\\rho_{ij}-1)h_{ij}$, so any budget rule that ignores pair inclusion cannot target the full leave-one-out gradient.","Marginal per-rollout propensities are insufficient; the ablations show marginal-only weighting reaches 47.6% accuracy and MSE 0.195, while joint inclusion reweighting reaches 49.2% and 0.145.","On a frozen candidate population, PAIR recovers near-zero bias (1.9e-3) at matched suffix cost, confirming that the correction mechanism itself, not just the learned predictor, restores the target.","At compute-matched budgets PAIR surpasses full-group GRPO and the strongest pointwise allocator on both backbones: +1.2 and +1.4 average accuracy over DUET at 0.49 and 0.48 of GRPO's token budget.","Self-normalized weights reduce gradient MSE to 0.139 but introduce explicit bias, illustrating that unbiasedness and variance are traded off rather than jointly optimized."],"supporting_citations":[{"why":"Establishes that the unclipped group-relative policy gradient is a U-statistic, defining the target PAIR estimates.","marker":"Zhou et al., 2026"},{"why":"Introduces the U-statistic class used to identify the pair-averaged target.","marker":"Hoeffding, 1948"},{"why":"Supplies the unequal-probability inclusion-weighting principle that PAIR extends to joint pair inclusion.","marker":"Horvitz and Thompson, 1952"},{"why":"Frames design-based inference for finite-population totals under joint inclusion probabilities.","marker":"Särndal et al., 1992"},{"why":"DUET is the strongest pointwise allocator baseline whose average-accuracy results PAIR is compared against.","marker":"Hu et al., 2026"},{"why":"Motivates the vertex-cost/edge-observation induced-subgraph geometry used for pair estimation.","marker":"Klusowski and Wu, 2018"},{"why":"Provides the fixed-size conditional-Poisson design whose joint inclusion probabilities must be computed rather than factored.","marker":"Hájek, 1964"}],"fun_headline_variants":["PAIR reweights rollouts pairwise, halving RLVR tokens","Pairwise inclusion reweighting halves RLVR compute","PAIR: unbiased rollout allocation with 50% fewer tokens","PAIR treats rollouts as graph edges to halve RLVR tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The exact design-unbiasedness guarantee holds only for the unclipped, unstandardized, on-policy leave-one-out target with probabilities logged before outcomes are seen, and the paper relies on that guarantee carrying through the practical pipeline that adds PPO clipping, reward standardization, and learned prefix predictors.","fun_headline_variants_meta":{"raw":{"variants":["PAIR reweights rollouts pairwise, halving RLVR tokens","Pairwise inclusion reweighting halves RLVR compute","PAIR: unbiased rollout allocation with 50% fewer tokens","PAIR treats rollouts as graph edges to halve RLVR tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1551,"prompt_tokens":1117,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":733,"completion_tokens_details":{"reasoning_tokens":362}},"tokens_in":733,"tokens_out":434,"duration_ms":4755,"temperature":1.0,"reasoning_tokens":362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:14:50.507105+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the frozen-population audit with a lookahead design: log inclusion probabilities before terminal rewards are observed, then reweight with probabilities recomputed after seeing rewards, so the pre-outcome logging condition of Theorem 2 is violated. If this post-outcome design keeps PAIR's near-zero bias (1.9e-3) and its reported accuracy gains, the theorem's central assumption is not what produces the empirical result; if bias appears, the assumption is load-bearing.","supporting_citations":[],"review_version":1}