{"id":"b003e4d7-c578-41a4-969c-1392775fe6be","arxiv_id":"2603.09692","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Uncertainty-guided active selection of large quality-gap response pairs produces RLHF preference data that matches or exceeds static baselines at roughly one-sixth the annotation budget.","lead":"ActiveUltraFeedback is an active-learning pipeline that uses reward uncertainty to pick which LLM response pairs to annotate for preference training. It reports matching or beating static preference datasets with as little as one-sixth the labels, mainly via two new gap-focused selectors (DRTS and DeltaUCB).","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Sample-efficiency claim is measured under a fixed synthetic judge; transfer to human annotation budgets is untested.","rationale":"The reader correctly isolates the judge–human transfer assumption as the weakest link supporting the strongest claim. The paper’s internal evidence is strong under the synthetic protocol: multi-prompt-source ablations (Section 5.4), multi-optimizer ablations (Section 5.5), open code/data, and clear demonstration that classic dueling-bandit objectives are misaligned with preference-data quality (Section 5.2, G.1). Those results stand. The load-bearing gap is that the efficiency numbers that justify “one-sixth of the annotated data” and the abstract’s low-resource/expert-domain motivation are measured only under the same LLM judge that also supplies the acquisition signal. Because DRTS/DELTAUCB are explicitly optimized for large predicted deltas, any systematic judge–human disagreement concentrated on those pairs would selectively erode the claimed gains. The concrete human re-annotation test above would settle the issue without requiring a full re-run of the pipeline. No stronger internal inconsistency appears; the concern is external validity of the efficiency claim, not correctness of the reported synthetic experiments. Verdict therefore remains CONDITIONAL, matching the reader.","tokens_in":38713,"tokens_out":696,"duration_ms":7817,"concrete_test":"On a stratified 1–2k-prompt subset of UltraFeedback, re-annotate the pairs already chosen by DRTS, DELTAUCB, RANDOM and UltraFeedback-style selection with human raters (same 4 aspects or overall preference). Retrain DPO and the reward model on the human-labeled subsets of equal size and replot the sample-efficiency curves of Figure 3. If the 5k–10k advantage of DRTS/DELTAUCB shrinks below the paper’s significance thresholds (0.008 downstream / 0.02 RewardBench 2) relative to RANDOM, the headline efficiency claim does not transfer.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (comparable/superior performance with ~1/6 the annotated pairs) is established entirely under Qwen3-235B-A22B aspect-wise probabilistic scoring (Section 4.4, Appendix E). DRTS and DELTAUCB deliberately select large predicted quality-gap pairs (Section 4.3; Table 23: rejected means ~1.97–2.11 vs. ~3.5–4.7 for RANDOM/DTS). The paper’s own analysis shows that regret-minimizing methods fail precisely because they produce small-gap pairs. If the judge’s ranking of those large-gap pairs systematically diverges from human ranking—especially on the hard, high-delta pairs the methods target—then the pairs that look most informative under the synthetic label may be less informative (or differently ordered) under human labels. Appendix E reports judge–human agreement only on AlpacaEval-style and RewardBench-2 proxies, not on the actively selected UltraFeedback-style pairs that drive the efficiency curves in Figures 3–5. Consequently the reported 5k–10k vs 60k sample-efficiency gains need not transfer to real human annotation budgets, which is the setting the abstract and introduction advertise.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces ACTIVEULTRAFEEDBACK, a modular active-learning pipeline for preference-data generation framed as a contextual dueling bandit. For each prompt, responses are drawn from a diverse 30-model pool, an Epistemic Neural Network (ENN) supplies reward means and uncertainties, and a pair is selected for annotation; the ENN is then updated. The authors systematically compare passive heuristics (RANDOM, MAXMIN, UltraFeedback-style, DeltaQwen), standard dueling-bandit rules (InfoMax, DTS, MaxMinLCB), and two novel delta-oriented rules (DRTS, DeltaUCB) that target large predicted quality gaps. Across UltraFeedback, Skywork, Combined, and Tulu-3 prompts, and under DPO, IPO, and SimPO, DRTS and DeltaUCB produce datasets that match or beat static baselines on GSM8K, IFEval, TruthfulQA, AlpacaEval 2, and RewardBench 2, often with roughly one-sixth the annotated pairs and only one comparison per prompt. Code and datasets are released.","tokens_in":39092,"tokens_out":1089,"duration_ms":10863,"significance":"If the sample-efficiency claims hold under realistic annotation, the work would materially lower the cost of preference data for RLHF, especially outside the training domains of fixed model families. Strengths include a clean modular design, multi-prompt and multi-optimizer ablations, seed/stability analyses (Appendix F.2), dataset diagnostics that explain why regret-minimizing rules underperform (Table 23, Figure 6), and full open-sourcing of pipeline, datasets, and models. The introduction of DRTS and DeltaUCB as practical, delta-aware acquisition functions is a concrete contribution that other labs can adopt immediately.","major_comments":[{"comment":"The central sample-efficiency claim (comparable/superior performance with ~5k–10k vs 60k pairs; Abstract, §5.3, Figures 3–5) is established entirely under Qwen3-235B-A22B aspect-wise probabilistic scoring (§4.4, Appendix E). DRTS/DeltaUCB deliberately select large-gap pairs (Table 23: rejected means ~1.97–2.11). Appendix E validates the judge only on AlpacaEval-style and RewardBench-2 proxies, not on the actively selected UltraFeedback-style pairs that drive the efficiency curves. Without a human-annotation pilot or judge–human agreement study on those high-delta pairs, transfer of the reported 1/6 annotation savings to real human budgets remains an untested assumption that is load-bearing for the abstract’s claim.","section":null},{"comment":"§5.2–5.3 and Table 2 show that DeltaQwen is competitive for DPO downstream scores yet collapses on RewardBench 2 and on IPO/SimPO (§5.5, Figure 5). The paper attributes this to limited diversity, which is plausible, but the practical recommendation (“use DRTS or DeltaUCB”) would be stronger if the authors quantified how much of the DPO gain of DeltaQwen is AlpacaEval-driven (they already note this in Appendix G.2) and stated more clearly when a static size-gap heuristic remains preferable. This is not a correctness error, but it affects how readers should interpret the “best method” claim.","section":null}],"minor_comments":[{"comment":"Figure 1 and the abstract claim “one-sixth of the annotated data”; the precise comparison (5k–10k vs 60k on UltraFeedback prompts under DPO) should be stated once in the main text with the exact baseline so readers do not have to reverse-engineer it from the figures.","section":null},{"comment":"Notation for UCB/LCB probabilities (Eqs. 3–4) re-uses pϕ for both; a short clarifying sentence or distinct symbols would reduce ambiguity when reading Algorithms 1–5.","section":null},{"comment":"Table 2 and Figure 3 report mean deltas; adding standard errors from the seed studies already present in Appendix F.2 to the main tables would make significance claims easier to assess at a glance.","section":null},{"comment":"The compute discussion (§5.3, F.4) correctly notes that pre-computing all responses inflates upfront cost; a short forward-looking paragraph on model-pool pruning (already partially supported by Table 26) would help practitioners estimate end-to-end cost.","section":null},{"comment":"Minor typographical inconsistencies appear in author affiliations and in the repeated “ACTIVEULTRAFEEDBACK” capitalization; a final pass would improve polish.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The work is solid empirical systems research with unusually thorough ablations and open artifacts. The only load-bearing gap is the synthetic-judge assumption for the sample-efficiency headline; a small human pilot or an explicit limitation statement would bring the paper comfortably over the accept bar for a top ML venue. I do not see novelty or citation issues that would warrant rejection."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: if you care about preference-data quality under a fixed prompt pool and a strong LLM judge, DRTS and DeltaUCB give clearer chosen–rejected gaps than random, UltraFeedback-style, or classic dueling-bandit selectors, and the resulting DPO/IPO/SimPO and reward-model curves look better at 5–10k pairs than the static baselines do at ~60k. That is the paper’s actual contribution.\n\nWhat is new is not “active RLHF” in the abstract—DTS, InfoMax, MaxMinLCB, and ENN-based selection already exist—but a single modular loop that runs the same selectors on both reward modeling (RewardBench 2) and three preference optimizers, plus two deliberately delta-oriented rules (DRTS: Thompson best vs reversed-Thompson worst; DeltaUCB: max optimistic preference probability). The diagnostics in the appendix are honest: regret-minimizing methods correctly find strong answers but pair them with other strong answers, so the supervision signal is weak; the delta methods keep high chosen scores and low rejected scores. Multi-prompt-source ablations (UltraFeedback, Skywork, Combined, Tulu-3) and open code/data make the comparison reproducible. Stability numbers are reported and look tight enough for the claims they make.\n\nSoft spots, in proportion. The central efficiency claim is measured entirely under Qwen3-235B aspect-wise probabilistic scoring. The methods are designed to surface large-gap pairs; if that judge’s ranking of those pairs diverges from humans, the 1/6 figure need not transfer to real annotation budgets. Appendix E gives proxy agreement numbers, not agreement on the actively selected pairs that drive Figures 3–5. Second, they pre-compute the full response pool and all judge scores for ablations, so the reported gains are annotation efficiency, not end-to-end compute; they say this themselves. Third, classic bandit objectives are shown to be misaligned with preference-data quality—that is a useful negative result, not a flaw. Hyperparameters (β, ensemble size, ζ decay, etc.) are free but swept and documented.\n\nThis is for people who build preference datasets or run active RLHF loops. It is not a theory paper. I would send it to peer review: the experimental design is careful, the negative result on regret methods is clean, and the open artifacts let others stress-test the judge-transfer question. Engage with it if you collect preference data; treat the human-label transfer as the open item, not as a reason to ignore the rest.","headline":"Solid modular active-preference pipeline with two new delta-oriented selectors that beat static and classic bandit baselines under a fixed synthetic judge; the 6× annotation claim is real in that setting but untested with humans.","tokens_in":39707,"tokens_out":642,"would_cite":true,"duration_ms":7644,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Active pair selection that targets quality gaps can match or beat static preference data with one-sixth the annotations.","keywords":["active learning","preference data","RLHF","dueling bandits","Delta Learning","DRTS","DeltaUCB","sample efficiency"],"falsifier":"Collect a modest human preference set on the same actively selected pairs and re-run the reward-model and DPO/IPO/SimPO evaluations; if human labels erase the sample-efficiency advantage over random or UltraFeedback-style pairs, the central efficiency claim fails under the intended annotation regime.","tokens_in":39635,"feed_emoji":"🎯","tokens_out":831,"duration_ms":8523,"temperature":0.7,"pith_summary":"Preference data for aligning language models is expensive, especially when experts must label it. This paper argues that you do not need to label every generated response pair: if you keep a running probabilistic model of response quality and deliberately pick the pairs that look most informative, you can build better datasets with far fewer labels. The authors package this idea as ActiveUltraFeedback, a modular loop that generates many candidate answers, estimates rewards and uncertainty, chooses a pair, obtains a preference label, and updates the reward model. Two new selection rules, Double Reverse Thompson Sampling and DeltaUCB, specifically chase large predicted quality gaps rather than only minimizing regret or uncertainty. Across several prompt collections and preference-tuning algorithms, those rules produce datasets that match or beat static baselines (including the original UltraFeedback pairs) while using as little as one-sixth of the annotations and only one comparison per prompt.","feed_headline":"Preference data cut to one-sixth with gap-seeking pairs","feed_subtitle":"Active selection that targets large quality gaps matches static baselines with far fewer labels","key_machinery":"Active delta learning via DRTS and DeltaUCB: selection rules that use ensemble-based upper and lower confidence bounds on reward to deliberately pair high- and low-quality responses rather than merely minimize dueling-bandit regret.","core_discovery":"ActiveUltraFeedback with the new DRTS and DeltaUCB selectors produces preference datasets that yield comparable or superior reward-model and downstream fine-tuning performance using as little as one-sixth of the annotated pairs relative to static baselines, while needing only a single pairwise comparison per prompt and remaining usable across prompt sources and optimization algorithms.","pith_inferences":["If response-generation cost later becomes the bottleneck, the same delta objective could be pushed upstream: choose which models to query rather than which already-generated answers to label.","Judge–human disagreement is likely largest precisely on the high-delta pairs that the new methods prefer, so calibration studies on those pairs would be the highest-leverage next measurement.","The modular loop invites testing whether simpler uncertainty estimators or smaller response pools can preserve most of the efficiency gain at lower compute."],"forward_implications":["When annotation budget is the bottleneck, DRTS or DeltaUCB become strong default pair selectors for preference data collection.","Selecting the single best response is not the same as selecting the most informative preference pair; large quality gaps give a cleaner training signal than near-ties between strong answers.","The same actively collected datasets can support both reward modeling and multiple direct preference optimizers without being locked to one model family.","Domains that lack large static preference corpora become more feasible once only a fraction of the usual labels are required."],"fun_headline_variants":["Active selection cuts preference labels to one-sixth","Gap-seeking pairs match baselines with far fewer annotations","DRTS and DeltaUCB yield equal quality on 1/6 of pairs","ActiveUltraFeedback needs one comparison per prompt for full gains","Uncertainty picks large-gap pairs to slash annotation costs"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"The pipeline treats an LLM judge’s aspect-wise scores as a faithful enough stand-in for the human preference signal that the method is meant to serve; if that proxy is systematically biased on the hard pairs that active selection chooses, the reported annotation savings may not transfer to real human budgets.","fun_headline_variants_meta":{"raw":{"variants":["Active selection cuts preference labels to one-sixth","Gap-seeking pairs match baselines with far fewer annotations","DRTS and DeltaUCB yield equal quality on 1/6 of pairs","ActiveUltraFeedback needs one comparison per prompt for full gains","Uncertainty picks large-gap pairs to slash annotation costs"]},"model":"grok-4.5","effort":"low","cost_usd":0.004398,"raw_usage":{"total_tokens":1209,"prompt_tokens":714,"num_sources_used":0,"completion_tokens":64,"cost_in_usd_ticks":43980000,"prompt_tokens_details":{"text_tokens":714,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":431,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":714,"tokens_out":64,"duration_ms":4008,"temperature":1.0,"reasoning_tokens":431,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-15T00:05:25.223879+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Collect a modest human preference set on the same actively selected pairs and re-run the reward-model and DPO/IPO/SimPO evaluations; if human labels erase the sample-efficiency advantage over random or UltraFeedback-style pairs, the central efficiency claim fails under the intended annotation regime.","supporting_citations":[],"review_version":1}