{"id":"52dd17b8-6193-4057-8969-c3a60142246d","arxiv_id":"2608.10905","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"ReOrder-OPD sorts OPD prompts by a proxy for teacher continuation reliability and consistently beats vanilla ordering on matched math and code experiments.","lead":"The paper proposes ordering prompts during on-policy distillation by how reliably the teacher can finish a student's partially generated answer, using a cheap one-rollout ROUGE-5 score as a proxy. In matched experiments across Qwen3 and Gemma4 models on math and code benchmarks, the reordered training improves aggregate scores by one to three points over vanilla distillation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is sound only if the deterministic verifier V is accurate and the training pools are uncontaminated; neither is audited, so V error or benchmark overlap could drive the ordering gains.","rationale":"Reader identified V accuracy and decontamination as the weakest assumption, and I agree that this is the most load-bearing concern. I considered the no-shuffle/fixed-order confound, since ReOrder disables shuffling while vanilla shuffles, but the paper's Table 4 and Figure 1(b) include ascending-R and other fixed deterministic orders and show that direction matters, so that confound is less decisive. The verifier threat is more fundamental: it invalidates not only the evaluation metric but the construction of the teacher library, the proxy score, and the oracle reliability measure itself. The absence of an audit is not proof of error, but the paper makes a strong universal claim on the basis of one unverified component, and the cost of auditing V and checking overlap is low. The result is plausible and internally consistent, and the reported evidence is substantial, so I keep the reader's CONDITIONAL verdict rather than escalating to rejection. My read does not change the reader's verdict; it reinforces the condition under which the paper should be accepted.","tokens_in":12555,"tokens_out":14887,"duration_ms":164315,"concrete_test":"Run two checks. First, verifier audit: take 200 final answers from DeepMath-5K teacher libraries and 200 from each of AIME24/25/26 and HMMT25/26 student evaluations under seed 42; have two independent annotators judge correctness and compute V's agreement and false-positive/false-negative rates. Second, contamination check: run 13-gram substring overlap between all DeepMath-5K/17K prompts and the 183 evaluation problems; remove any overlapping problems and re-run Table 1 for all five students at seed 42 on the clean subset. If V agreement is at least 99% and the six-set gains remain positive after overlap removal, the concern is resolved; otherwise, the ordering result must be re-benchmarked under a corrected verifier or decontaminated pool.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every component of the pipeline inherits V: Eq. 6 retains only V-correct teacher trajectories, Eq. 7 scores a student rollout by similarity to those trajectories, Eq. 5 defines oracle R through V-accepted continuations, and the mean@16 evaluation itself uses V. The paper calls V deterministic and says it checks final answers, but reports no error audit on DeepMath-5K/17K or Eurus code subsets and no decontamination check against AIME24-26/HMMT25-26. If V has false positives, wrong solutions enter C_t and can inflate q for wrong student rollouts; if V has false negatives, valid references are discarded and prompts fall into the unscored tail, changing which prompts are visited within the fixed update budget. If evaluation problems overlap the training pool, q can up-rank memorized prompts, so ReOrder's early exposure of those prompts would inflate the measured gains relative to shuffled vanilla even if ordering has no general value. These threats are not merely hypothetical: the paper's own limitation statement acknowledges that matched comparisons do not control preprocessing cost, and no code or data release permits an independent audit. The central claim that ordering improves every matched aggregate therefore rests on an unverified assumption about V and dataset separation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReOrder-OPD, a prompt-scheduling layer for on-policy distillation (OPD). It defines prompt-level teacher continuation reliability R as the verifier-evaluated probability that a teacher can continue a student-generated prefix to a correct answer, and then introduces a one-rollout proxy q: the maximum ROUGE-5 F1 between a single fresh student rollout and verifier-correct teacher responses for the same prompt. Static ReOrder-OPD sorts the fixed prompt pool by q before running vanilla OPD, and a dynamic variant re-sorts unvisited prompts a few times during training. The authors report oracle experiments on DeepMath-1K showing that high-R prompts yield larger OPD gains and that descending-R order beats vanilla and ascending-R order; they validate q by showing that mean R rises monotonically across ten equal-frequency q bins. Main experiments on DeepMath-5K/17K and a Eurus code subset report matched aggregate gains over vanilla OPD for five student models, positive aggregate gains in all FiRe-OPD and ExOPD compositions, and gains at 1K/5K/17K scales.","tokens_in":12802,"tokens_out":5488,"duration_ms":57832,"significance":"If the central claim is sound, the paper identifies a useful and previously under-explored axis of intervention for OPD: when a prompt is visited, rather than how the sampled trajectory is supervised. The formal definition of R, the oracle analysis, the monotonic proxy validation, and the breadth of matched comparisons across two model families, five student scales, mathematics and code, and composition with two trajectory-level objectives are strengths. The paper is also unusually explicit about its limitations, acknowledging that matched comparisons do not equalize total generation and verification cost. However, the evidence as presented does not yet establish that the gains are due to reliability-based ordering: the treated condition receives extra preprocessing compute, data shuffling is disabled only in the treated condition, the verifier V is not audited, no decontamination check is reported against the evaluation benchmarks, and significance testing is absent. These gaps are load-bearing for the central claim, so the contribution is significant but conditional.","major_comments":[{"comment":"The claimed isolation of prompt order is not achieved because the treated condition receives additional computation and data. In 'Experimental Setup' the paper states that 'matched comparisons equalize prompt visits and update counts, not total generation and verification cost,' and the method description shows that ReOrder-OPD adds |D| student scoring rollouts and builds a teacher library of up to 16 verifier-correct responses per prompt (Eq. 6) that vanilla OPD does not have. Consequently Tables 1-3, 6, and 7 compare vanilla OPD against vanilla OPD plus an extra student-inference, teacher-generation, and verification budget, not against a condition that differs only in prompt order. Please add a control that spends the same extra budget on the vanilla baseline, for example by sampling one extra student rollout per prompt in a random or fixed order and training on those rollouts, and report whether the ordering gains persist when total generated and verified tokens are equalized.","section":"Experimental Setup and ReOrder-OPD"},{"comment":"The comparison also changes data-shuffling behavior. The paper says 'To preserve the order, ReOrder-OPD disables data shuffling; vanilla OPD shuffles randomly.' This means the treated condition differs from the baseline not only by sorting prompts by q but also by replacing random mini-batch order with a fixed deterministic order. Fixed-order SGD can itself change optimization dynamics, so the observed gains may partly reflect the removal of shuffling rather than the reliability ranking. Please include a control where vanilla OPD uses the same deterministic order but with prompts sorted in an order unrelated to q (for example, the original dataset order or a random fixed order), while keeping all other settings matched.","section":"ReOrder-OPD"},{"comment":"Every component of the pipeline inherits the correctness of the deterministic verifier V, but no verifier-error audit is reported. R in Eq. (3), the teacher library C_t in Eq. (6), the proxy q in Eq. (7), the oracle estimate in Eq. (5), and the mean@16 evaluation metric all depend on V. If V has false positives, incorrect teacher trajectories enter C_t and can inflate q for incorrect student rollouts; if V has false negatives, valid references are discarded and prompts fall into the unscored tail, changing which prompts are visited within the fixed update budget. No audit is reported on DeepMath-5K/17K or the Eurus-2-RL-Data code subset, and no decontamination check is reported between the training subsets and AIME24-26/HMMT25-26/HumanEval+/MBPP+/LCB-v6. The citation of DeepMath-103K as 'decontaminated' does not by itself cover the exact 5K/17K subsets or the code data used here. Please report (i) verifier accuracy on a human-labeled sample of teacher and student responses from each domain, and (ii) exact and near-duplicate overlap statistics between the training pools and the evaluation benchmarks.","section":"Equations (3)-(7) and Benchmarks and metrics"},{"comment":"The central empirical claim is supported mainly by aggregate mean differences over three seeds, but no significance tests or confidence intervals are reported. Several per-benchmark differences are within one seed standard deviation of zero, for example Table 1, Qwen3-8B AIME24 (59.51±1.58 vs 59.79±2.35) and Qwen3-4B HMMT25-Nov (35.83±1.27 vs 33.89±0.98), and Table 3 reports all FiRe-OPD/ExOPD compositions at a single seed with no uncertainty. To support the abstract's statement that ReOrder 'improves every matched aggregate comparison,' please add paired significance tests over seeds for the Six-set and Code Avg. aggregates, and either run Table 3 over multiple seeds or present bootstrap confidence intervals for the single-seed composition gains.","section":"Tables 1-3"}],"minor_comments":[{"comment":"The Spearman correlations in Figure 1(a) are reported without confidence intervals or p-values; with 905 prompts, adding these would help the reader judge the claim that local signals are only weakly associated with continuation success.","section":"Figure 1(a)"},{"comment":"The prefix-sampling rule G(y_s) is not formally specified in the main text; the experiments later state that five uniformly spaced nonterminal prefixes are used, but the definition of R in Eq. (4) should state the general rule or refer explicitly to the experimental instantiation.","section":"Equation (4)"},{"comment":"For the 'ROUGE-5 / random' row, the paper reports mean±SD over 20 repetitions but does not state how the random teacher trajectories are sampled; a one-sentence clarification would make the ablation reproducible.","section":"Table 5"},{"comment":"The dynamic refresh schedule uses fractions alpha_1 < ... < alpha_J of the queue consumed, but the experimental values in Table 7 (updates 133, 177, 222 out of 266) are only given later; moving this schedule specification into the Algorithm or its caption would improve readability.","section":"Algorithm 1"},{"comment":"The conclusion states that 'Verifier-correct libraries remain necessary,' but the cost of building these libraries is not quantified in FLOPs or wall-clock time; a short cost analysis would make the preprocessing trade-off more transparent.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not include a code or data release, which is a concern if the journal expects reproducibility for empirical claims of this kind. The single-seed composition table (Table 3) and the lack of significance tests are the main evidentiary weaknesses; the verifier and decontamination audits are essential before publication. The paper's core idea is interesting and the prose is clear, but the current evidence does not yet isolate prompt ordering as the causal factor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you work on OPD or curriculum-style ordering for reasoning distillation. The genuinely new piece is the prompt-level reliability statistic R, defined through teacher continuation outcomes, and the cheap one-rollout max ROUGE-5 proxy that tracks R's coarse strata. The oracle experiments (Table 4, Figure 3) are the paper's strongest evidence: high-R prompts give larger OPD gains, descending-R ordering beats random/ascending, and the proxy's decile bins rise monotonically with R. That is a real result, not just a heuristic dressed up.\n\nThe main empirical claim appears solid as far as the reported numbers go: consistent aggregate gains across two model families, five student scales, math, and code, and positive composition with FiRe-OPD and ExOPD. The matched comparisons in Tables 1 and 2 are appropriately designed, holding everything but prompt order fixed.\n\nSoft spots, in proportion: First, no significance tests anywhere. Several benchmark-level differences in Table 1 are within one seed standard deviation of zero; the aggregate gains are consistent, but I would have liked paired tests or at least seed-level CIs. Second, Table 3 uses a single seed for the composition study, which weakens that claim. Third, no code or data release, so the whole pipeline is un-auditable from the text alone. Fourth—and this is the skeptic's main point—the deterministic verifier V is load-bearing: it builds the teacher library, defines R, scores the proxy, and evaluates the benchmark, yet the paper reports no verifier-error audit and no decontamination check against the test sets. That is a legitimate concern, but it is not fatal: if V were badly wrong, the evaluation itself would be untrustworthy, and the consistent multi-seed ordering effect suggests the method is doing something real. Still, an audit of V on the actual subsets would settle it. The paper's own note that preprocessing cost is not matched between vanilla and ReOrder is an honest limitation, not a hidden flaw.\n\nThis deserves a serious referee. The work is new, the motivation is clear, and the evidence is mostly consistent. I would send it to review with a request for significance tests, a verifier-error audit, and at least one additional seed for the composition study. If those are added, I would cite it.","headline":"A useful, well-organized prompt-ordering layer for OPD with consistent reported gains; the main gaps are missing significance tests and an un-audited verifier, both fixable.","tokens_in":13373,"tokens_out":1709,"would_cite":true,"duration_ms":19849,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reordering prompts by teacher-continuation reliability lifts on-policy distillation in every tested setting.","keywords":["on-policy distillation","teacher continuation reliability","prompt ordering","curriculum learning","ROUGE-5 proxy","verifier-correct teacher library","mathematical reasoning","code generation"],"falsifier":"Run ReOrder-OPD on a prompt set where the verifier is deliberately corrupted (for example, flipping a known fraction of correctness labels), or on a benchmark that overlaps the training pool without decontamination; if the matched aggregate gains vanish or reverse in either case, the central claim that the proxy recovers reliability-based ordering is falsified.","tokens_in":12346,"feed_emoji":"📈","tokens_out":2745,"duration_ms":24724,"temperature":0.7,"pith_summary":"The paper argues that in on-policy distillation (OPD), where a teacher supervises a student on the student's own sampled trajectories, the order in which prompts are visited matters as much as how each trajectory is supervised. It defines prompt-level teacher continuation reliability R: the probability that the teacher, starting from a student-generated prefix, reaches a correct final answer. Oracle experiments show that training on high-R prompts first yields larger gains, and that simply sorting a fixed prompt pool by a cheap proxy for R improves distillation consistently across model families, math benchmarks, and code generation. The proxy requires one student rollout per prompt and a reusable library of verifier-correct teacher solutions, making it inexpensive to apply.","feed_headline":"Reordering prompts lifts distillation in every tested setting","feed_subtitle":"Sorting a fixed pool by a one-rollout ROUGE-5 proxy improves math and code gains for Qwen3 and Gemma4.","key_machinery":"The load-bearing object is prompt-level teacher continuation reliability $R(x;\\pi_s,\\pi_t)$, defined as the expected probability that the teacher completes a student-generated prefix to a verifier-accepted answer, averaged over prefixes and trajectories induced by the current student policy. The practical proxy is a one-rollout score $q(x;\\pi_s)=\\max_{y^t\\in C_t(x)} \\mathrm{ROUGE\\text{-}5\\,F1}(\\tilde{y}^s(x;\\pi_s), y^t)$, computed against a fixed library $C_t(x)$ of verifier-correct teacher responses. The argument that order matters rests on the non-commutativity of OPD updates: presenting prompt $a$ before prompt $b$ changes the policy that generates the trajectory for $b$, so early exposure to high-reliability prompts propagates through training.","core_discovery":"ReOrder-OPD sorts the fixed prompt pool by the maximum full-response ROUGE-5 F1 between one independent student rollout and verifier-correct teacher trajectories for the same prompt, then runs vanilla on-policy distillation with fresh on-policy rollouts, disabling data shuffling to preserve the order. The paper claims this ordering change alone raises the matched Six-set aggregate for all five tested student–teacher configurations (Qwen3-1.7B/4B/8B and Gemma4-E2B/E4B) by 1.09–2.58 percentage points, improves all six student–seed code aggregate comparisons, and adds gains on top of FiRe-OPD and ExOPD in all six tested combinations. The core evidence is that the one-rollout ROUGE-5 score partitions prompts into deciles whose mean diagnostic R rises monotonically from 0.29 to 0.98, so the proxy preserves the coarse reliability structure needed for scheduling, while local signals like teacher log-probability, top-k overlap, and step divergence do not.","pith_inferences":["The maximum-over-correct-solutions aggregation suggests a testable extension: using the same proxy to select prompts for rejection sampling or to weight prompts by reliability during offline distillation, not just ordering them.","Because $R$ measures teacher continuation success from student-visited states, the proxy could serve as a cheap diagnostic for teacher–student alignment before committing to a distillation run.","A verifier-noise audit would sharpen the claim: if $V$ mislabels answers, the teacher library and the proxy inherit the errors, so the method's robustness to verifier accuracy is an open question the paper does not address."],"forward_implications":["Prompt ordering is an intervention level orthogonal to trajectory-level supervision; ReOrder can be composed with methods that filter, reweight, or gate within a rollout, as shown by gains on FiRe-OPD and ExOPD.","The ordering benefit transfers from mathematical reasoning to code generation with identical methodology, suggesting the reliability-aware scheduling principle is not domain-specific.","Dynamic refresh, which re-sorts only the unvisited queue using current-student rollouts, beats static ordering at the 17K-prompt scale, indicating that the optimal priority evolves with the student policy.","The proxy is cheap: it requires one student rollout per prompt and a teacher library that is reused across students, seeds, and objectives, but no teacher continuations during scoring."],"supporting_citations":[{"why":"Supplies the ROUGE metric used to define the one-rollout proxy in Eq. 7.","marker":"Lin 2004"},{"why":"Establishes the basic form of on-policy distillation (GKD) that ReOrder-OPD builds on.","marker":"Agarwal et al. 2024"},{"why":"Another on-policy distillation baseline (MiniLLM) that computes teacher supervision on student trajectories.","marker":"Gu et al. 2024"},{"why":"Provides FiRe-OPD, the trajectory-level filtering and reweighting method that ReOrder is composed with.","marker":"Li et al. 2026b"},{"why":"Provides ExOPD, the objective-side distillation method that ReOrder is composed with.","marker":"Yang et al. 2026a"},{"why":"Source of DeepMath-1K/5K/17K prompt pools used for reliability studies, main comparisons, and scaling.","marker":"He et al. 2025"},{"why":"Source of the Eurus-2-RL-Data code subset used for code-domain validation.","marker":"Cui et al. 2025"},{"why":"Provides the EvalPlus harness (HumanEval+ and MBPP+) used to measure code correctness.","marker":"Liu et al. 2023"},{"why":"Provides LiveCodeBench V6, one of the code evaluation benchmarks.","marker":"Jain et al. 2024"}],"fun_headline_variants":["Reorder prompts by proxy to gain 1-3 points in distillation","Reliability-aware prompt ordering lifts every tested setting","One-rollout proxy sorts prompts for better on-policy distillation","Prompt order alone: up to 2.58 points gain in OPD","Sorting prompts by ROUGE-5 proxy lifts math and code distillation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The deterministic answer verifier $V$ correctly identifies whether a final answer is right, and every derived quantity (teacher library, proxy, and measured gains) inherits whatever mistakes $V$ makes.","fun_headline_variants_meta":{"raw":{"variants":["Reorder prompts by proxy to gain 1-3 points in distillation","Reliability-aware prompt ordering lifts every tested setting","One-rollout proxy sorts prompts for better on-policy distillation","Prompt order alone: up to 2.58 points gain in OPD","Sorting prompts by ROUGE-5 proxy lifts math and code distillation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000969,"raw_usage":{"total_tokens":4171,"prompt_tokens":1042,"completion_tokens":3129,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":3038}},"tokens_in":658,"tokens_out":3129,"duration_ms":24626,"temperature":1.0,"reasoning_tokens":3038,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:34:11.612920+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ReOrder-OPD on a prompt set where the verifier is deliberately corrupted (for example, flipping a known fraction of correctness labels), or on a benchmark that overlaps the training pool without decontamination; if the matched aggregate gains vanish or reverse in either case, the central claim that the proxy recovers reliability-based ordering is falsified.","supporting_citations":[],"review_version":1}