{"id":"1ac4d031-b9cb-49c7-a4e3-8ebd3f574684","arxiv_id":"2411.10914","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"BPO balances knowledge breadth and depth in preference data by compressing prompts and dynamically augmenting the number of response pairs per prompt using gradient-based clustering, achieving stronger alignment with roughly 10% of the data.","lead":"This paper proposes BPO, a method that keeps only the most representative prompts from an alignment dataset and adds extra response pairs only for the prompts where the model learns most. The authors report that BPO matches or beats baselines that use ten times more data on MT-Bench and AlpacaEval, though the results lack error bars and code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's own data sizes contradict the 'no more than 10% overall data' claim: BPO uses 3,312 vs 27,000 pairs (12.3%) on HH-RLHF with Llama-3-8B and 2,753 vs 27,000 (10.2%) with Llama-2-13B; the central data-efficiency claim needs a corrected accounting or a strict cap.","rationale":"The paper proposes a plausible pipeline and reports consistent gains across SafeRLHF, HH-RLHF, and UltraFeedback; the random-selection ablation in Figure 6 and Table 5 give some internal evidence that the selection heuristic does something. I do not question the authors' honesty. My concern is narrower: the central claim is about data compression, and the paper's own Table 1 does not support the 'no more than 10% overall data' wording for the HH-RLHF rows under the natural pair-count reading. The pipeline arithmetic in §3.2 plus Appendix H is ambiguous about how many candidate pairs are formed from the 16 generated responses, and the reported Data Size column cannot be reproduced without an additional convention. This must be resolved before the efficiency claim can be evaluated. Secondary concerns (single seed, hyperparameters chosen on MT-Bench, no released code) reinforce the conditional verdict but are not the load-bearing issue. If the authors provide data-flow counts and show BPO wins under a strict 10% cap, I would accept; if not, the headline claim should be revised.","tokens_in":15313,"tokens_out":13743,"duration_ms":142940,"concrete_test":"Request the exact data-flow counts for one configuration (Llama-3-8B, HH-RLHF): |D_org|, |X_rep| after Eq. (1), number of generated responses, number of candidate pairs after score-difference filtering, the pair-formation rule (15 vs 120 pairs per prompt), and |D_dyn| after Eq. (6). Then compute r = |D_dyn| / 27,000. If r > 0.10 for any reported row, re-run BPO with a hard cap of 2,700 final pairs (10% of the vanilla DPO budget) and report MT-Bench/AlpacaEval with at least 3 seeds. If BPO no longer beats RS-DPO and Curry-DPO under that cap, the headline should be revised; if the gains persist, the claim should be reworded to specify that the 10% applies to selected prompts, not to final preference pairs.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"BPO's headline is a data-efficiency claim: 'our method BPO can achieve comparable or even better performance with no more than 10% overall data' (Table 1 caption). This is not consistently true under the natural reading of the table. In the HH-RLHF block, vanilla DPO trains on 27,000 preference pairs; BPO reports Data Size 3,312 for Llama-3-8B and 2,753 for Llama-2-13B, i.e., 12.3% and 10.2% of the vanilla budget. Even the KBC (s=10%) row reports 2,744 pairs, slightly above 2,700. If instead 'overall data' means the full 170K HH-RLHF set, then the comparison is not controlled: the vanilla baseline itself uses only a 27K subsample, so the 10% budget is not defined against the same reference. The pipeline description also makes the accounting hard to verify: KBC selects 10% of prompts, k=16 responses per prompt are generated, score-difference filtering keeps 'the top 10% of pairs', and Eq. (6) then keeps η=10% per gradient cluster. Depending on how many preference pairs are formed from the 16 responses (15 if one chosen response is paired with each rejected response, or 120 if all ordered pairs are used), the chained final size varies from roughly 1.5% to 12% of the original pair count; the paper does not state which convention produces the Data Size column. Thus the central '≤10% data' claim cannot be checked from the paper as written. The gradient-clustering mechanism is also asserted, but it is less decisive here because Figure 6's random-selection comparison gives it some internal support; the budget inconsistency is directly falsifiable from the paper's own numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces knowledge breadth and knowledge depth as complementary notions for alignment data, arguing that preference datasets over-represent breadth (many prompts, two responses each) and under-represent depth. It proposes a simple balancing method (Knowledge Breadth Compression plus Knowledge Depth Augmentation) and then a full method, Balanced Preference Optimization (BPO), which selects a representative subset of prompts via embedding clustering, generates multiple candidate responses per prompt with an SFT policy, scores them with GPT-4, and dynamically allocates more response pairs to prompts whose gradient features lie near cluster centroids after random projection of LoRA gradients. The final DPO training is performed on the selected pairs. Experiments on SafeRLHF and HH-RLHF with Llama-3-8B and Llama-2-13B, plus an additional UltraFeedback experiment, report that BPO outperforms vanilla DPO, Curry-DPO, and RS-DPO on MT-Bench and AlpacaEval while using roughly 10% of the training data, and ablations compare clustering algorithms, cluster counts, and alternative depth-measurement signals.","tokens_in":15761,"tokens_out":4175,"duration_ms":42973,"significance":"If the central data-efficiency claim holds, BPO would be a practically valuable result: it would show that aggressive prompt selection plus per-prompt depth augmentation can match or beat full-data preference optimization at a fraction of the compute. The paper has real strengths: it tests two model families and multiple benchmarks, includes an extra UltraFeedback evaluation, provides detailed ablations of clustering choices and depth measures, and candidly acknowledges in the Limitations section the reliance on GPT-4 scoring and the limited gains from pure self-optimization. The code release is promised, which would help reproducibility. However, the evidence as presented does not yet establish the headline claim: the main tables contain no error bars or significance tests, the reported data sizes in Table 1 do not consistently satisfy the 'no more than 10%' statement, and the gradient-centroid informativeness mechanism is asserted rather than validated against matched random selection with statistical controls. These issues are fixable in revision, but they are load-bearing for the paper's central contribution.","major_comments":[{"comment":"The headline claim that BPO uses 'no more than 10% overall data' is not supported by the Data Size column under the natural reading. On HH-RLHF, BPO reports 3,312 pairs for Llama-3-8B and 2,753 pairs for Llama-2-13B, while vanilla DPO uses 27,000 pairs; these are 12.3% and 10.2% of the vanilla budget, respectively. The KBC (s=10%) row also reports 2,744 pairs, slightly above 10% of 27,000. If 'overall data' instead refers to the full 170K HH-RLHF set, then the comparison is uncontrolled because the vanilla baseline itself trains on only a 27K subsample. The paper should state the reference set explicitly, report exactly how many preference pairs are constructed from the k=16 generated responses before the 'top 10% of pairs' filter (Appendix H), and either cap the data budget at 10% of the same reference or revise the claim.","section":"Table 1 and Section 4.2"},{"comment":"The statement that selecting only 1% to 10% of prompts via embedding clustering 'can achieve performance comparable to, and even surpassing, alignment on the full dataset' is contradicted by the HH-RLHF rows in Table 4. At a 10% scaling ratio, MT-Bench is 4.11 versus 4.71 for the full set, and at 20% the AlpacaEval win rate is 16.77% versus 18.14% for the full set. The claim should be restricted to the specific settings where the result actually holds, or rephrased as a mixed outcome that depends on dataset and metric.","section":"Section 2.3 and Appendix A, Table 4"},{"comment":"All reported comparisons are single-run point estimates without error bars, confidence intervals, or significance tests. Several differences are small enough to be within evaluation noise: for example, on SafeRLHF with Llama-2-13B, MT-Bench is 5.85 for BPO versus 5.79 for RS-DPO, and on HH-RLHF with Llama-2-13B, AlpacaEval is 26.96% for BPO versus 25.26% for RS-DPO. The paper should report multiple seeds with means and standard deviations, or otherwise quantify variability, for the main tables before claiming consistent superiority.","section":"Section 4.2, Tables 1, 5, and 6"},{"comment":"The core mechanism of BPO is that gradient vectors closest to cluster centroids mark the most informative samples and therefore deserve more depth resources. This is an asserted heuristic, and the supporting evidence is Figure 6 plus final benchmark scores that were also used to choose the number of gradient clusters and the selection ratio eta (Figure 5 and Table 5). Because G and eta are tuned on the same evaluation benchmarks, the improvement over random selection could partly reflect selection on the test set. I recommend adding a matched-compute comparison with random selection across multiple seeds and, ideally, using a held-out validation split for hyperparameter selection, so the benefit of the gradient signal can be separated from tuning effects.","section":"Section 3.2, Eq. (6), and Section 5.1"}],"minor_comments":[{"comment":"There are numerous typos and duplicated words, including 'hierichical' (Section 4.2), 'conducteded' and 'anselectedcted' (Appendix I), 'obtian' (Section 3.2), and 'B BPO achieves' (Section 4.2). The paper needs a careful copyedit.","section":"Throughout"},{"comment":"The notation for the selection ratio is inconsistent: Eq. (1) defines the breadth-compression ratio as s, while the caption of Figure 5 says 'The top eta = 10% of data points are selected based on Equation 1.' Use one symbol for the proportion selected in each stage.","section":"Eq. (1) and Figure 5"},{"comment":"The pair-construction convention is not specified. From k=16 generated responses, one could form 15 pairs (one chosen response versus each rejected response) or 120 ordered pairs, and the final Data Size after the 'top 10% of pairs' filter depends strongly on this choice. This should be stated explicitly so the data-efficiency claim can be checked.","section":"Appendix H"},{"comment":"The caption says 'almost only 1% to 10% of the prompts can lead to promising outcomes,' but the HH-RLHF columns show a non-monotonic pattern, with 20% worse than 1% on AlpacaEval. The caption overstates the regularity of the trend.","section":"Appendix A, Table 4"}],"recommendation":"major_revision","confidential_remarks":"This is not a reject: the empirical results are promising and the limitations section is candid. The main issues are that the headline data-efficiency claim is overstated relative to Table 1, the preliminary '1% to 10% matches full data' claim is contradicted by the paper's own Appendix A numbers, and the absence of error bars makes the performance gaps hard to interpret. I would invite a revision that fixes the accounting, adds statistical controls, and softens the claims to match the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: BPO is a genuinely new mechanism in a crowded field—dynamic per-prompt knowledge-depth augmentation via gradient clustering—and the experiments mostly support the method's usefulness. But the paper's headline data-efficiency claim is not true under its own Table 1. BPO uses 3,312 pairs vs 27,000 for vanilla DPO on HH-RLHF with Llama-3-8B (12.3%), and 2,753 (10.2%) with Llama-2-13B. The caption says 'no more than 10% overall data,' so the claim needs correction or the accounting needs a different reference point. This is the kind of thing that should be caught in revision.\n\nWhat's new: the breadth/depth framing, the simple balance experiment, and the gradient-based clustering to decide how many response pairs each prompt gets. The comparison against length and semantic similarity (Table 2) is a nice touch. Figure 6's random-selection comparison gives some internal support to the clustering, though it is not independent validation. The extra UltraFeedback results strengthen the empirical case. So the concept has legs.\n\nSoft spots: no error bars or multiple seeds anywhere; the MT-Bench gains are often within a few tenths, so I'd want to see stability. Section 2.3 overclaims: on HH-RLHF, 10% KBC gets MT-Bench 4.11 vs 4.71 full data (Appendix A), which directly contradicts the 'comparably or even surpassing' claim. The gradient-clustering informativeness assumption is asserted rather than derived, and the final scores were used to pick cluster counts and ratios; a held-out hyperparameter choice would help. The pair-construction convention (how many pairs are formed from 16 responses) is unspecified, making the final Data Size column hard to verify.\n\nNone of these are fatal to the method—the empirical pattern is fairly consistent across datasets and model sizes. But the paper needs a corrected data-size accounting, error bars, and a toned-down '≤10%' claim before it is citable as claimed.\n\nMy recommendation: send it to peer review. A serious referee can push for the fixes. The method is new enough and the evidence is promising enough to deserve that time.","headline":"BPO's dynamic per-prompt depth augmentation is a real idea, but the paper's '≤10% data' claim is contradicted by its own Table 1 and needs a fix before publication.","tokens_in":16324,"tokens_out":2821,"would_cite":true,"duration_ms":25887,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that balancing the number of prompts and response pairs in alignment data, then selecting the most informative pairs by their gradient signatures, lets a preference-optimized LLM match or beat standard DPO, RS-DPO, and…","keywords":["knowledge breadth","knowledge depth","preference data optimization","DPO","gradient clustering","data compression","alignment","rejection sampling"],"falsifier":"Replace the gradient-centroid selection in Eq. (6) with random selection of the same number of response pairs per prompt, keeping every other component fixed; if the random-selection run matches BPO's MT-Bench and AlpacaEval scores, then the claim that centroid proximity tracks informativeness is falsified. A second check is to train BPO on a held-out distribution (e.g., a different domain) and see whether the performance advantage over random depth allocation persists.","tokens_in":15113,"feed_emoji":"⚖️","tokens_out":4817,"duration_ms":40465,"temperature":0.7,"pith_summary":"The paper tries to show that alignment tuning data is imbalanced: many prompts but only two responses each, leading to over-learning of breadth and under-learning of depth. It proposes Balanced Preference Optimization (BPO), which first compresses prompts to a representative subset and then dynamically augments knowledge depth per prompt, using gradient-based clustering to pick the response pairs closest to cluster centroids. The central claim is that BPO with no more than 10% of the total data matches or exceeds vanilla DPO, RS-DPO, and Curry-DPO on MT-Bench and AlpacaEval, while reducing training cost. A reader should care because it suggests preference data can be aggressively compressed and selectively augmented without sacrificing alignment quality.","feed_headline":"BPO beats DPO using only 10% of preference data","feed_subtitle":"Balancing prompt count vs response pairs makes alignment cheaper and stronger.","key_machinery":"The central object is knowledge depth allocation driven by gradient-based clustering. Concretely: after SFT with LoRA, each candidate (prompt, response pair) is assigned a projected gradient vector $\\hat\\nabla\\ell(z;\\theta_t)$ obtained by random projection of the LoRA gradient; K-means groups these into clusters, and within each cluster the pairs closest to the centroid (smallest Euclidean distance to $\\mu_g$) are selected. This selection rule — Eq. (6) — is what converts \"informativeness\" into a concrete depth allocation: prompts with pairs near centroids get more response pairs, and the augmented set $D_{dyn}$ is then fed to DPO. The machinery also includes the embedding-based K-means prompt selection (Eq. 1) that compresses breadth by keeping the prompts nearest to prompt-embedding centroids.","core_discovery":"On the paper's own terms: we introduce knowledge breadth and knowledge depth as complementary measures of what an alignment dataset teaches, and we argue that the typical dataset structure — tens of thousands of prompts with only two responses each — allocates most learning to breadth and starves depth. We show that a simple uniform rebalancing, compressing prompts to 1–10% of the original set and augmenting the remaining prompts with extra response pairs, improves DPO results on SafeRLHF and HH-RLHF. Building on this, BPO makes depth augmentation dynamic: it generates multiple responses per prompt from an SFT policy, scores them with an LLM judge, and uses K-means on randomly projected LoRA gradients to select the response pairs closest to the gradient centroids, allocating more response pairs to prompts whose features sit near a centroid. We report that BPO with no more than 10% of the overall data achieves comparable or better MT-Bench and AlpacaEval scores than vanilla DPO, RS-DPO, and Curry-DPO across Llama-2-13B and Llama-3-8B, and that gradient-based depth allocation outperforms allocation based on response length or semantic similarity.","pith_inferences":["The paper's claim that centroid-adjacent gradient samples are more informative is a heuristic; it would be worth testing whether selecting the same number of pairs per prompt but uniformly at random from the gradient clusters (rather than from the centroids) collapses the performance gap, which would indicate the centroid rule is doing less work than claimed.","The 10% data compression suggests that most preference data may be redundant for DPO-style updates; if so, scaling laws for alignment data could be much shallower than scaling laws for pre-training.","A natural extension is to make the depth allocation adaptive during training, recomputing gradients and re-clustering after each epoch rather than once, which the paper does not explore.","The jailbreak-based response generation for safety data hints that depth augmentation only helps when the augmented pairs are actually separable (winning vs losing); a testable prediction is that the benefit of BPO shrinks when the judge scores have low variance."],"forward_implications":["If BPO is right, alignment pipelines can be run on roughly one-tenth of the original preference data, cutting training time and memory while keeping or improving benchmark wins.","The breadth/depth framing implies that simply increasing the number of responses per retained prompt is a reusable lever for alignment quality, independent of the base DPO objective.","Because gradient features, rather than surface statistics like response length or similarity, determine which samples get more depth, the method suggests that optimizing in gradient space is a better proxy for what the model needs to learn.","BPO's reported success on SafeRLHF, HH-RLHF, and UltraFeedback suggests the depth-allocation recipe transfers across safety, helpfulness, and general instruction-following data."],"supporting_citations":[{"why":"Supplies the base DPO objective that BPO uses for final alignment training after depth augmentation.","marker":"Rafailov et al., 2024"},{"why":"Defines the RS-DPO baseline and is the source of the rejection-sampling scheme used to generate contrastive response pairs.","marker":"Khaki et al., 2024b"},{"why":"Defines the Curry-DPO baseline using multiple ranked preference pairs per prompt, which BPO compares against and extends.","marker":"Pattnaik et al., 2024b"},{"why":"Provides the random projection of LoRA gradients and the gradient-based data selection that BPO adapts for dynamic depth allocation.","marker":"Xia et al., 2024"},{"why":"Supplies the 'less is more' evidence that a small subset of carefully selected prompts suffices for alignment, motivating knowledge breadth compression.","marker":"Zhou et al., 2024"},{"why":"Provides MT-Bench and the LLM-as-a-judge scoring that BPO uses to evaluate generated response pairs.","marker":"Zheng et al., 2023a"},{"why":"Provides AlpacaEval, the automatic evaluation benchmark used to measure win rates in the experiments.","marker":"Li et al., 2023"},{"why":"Supplies LoRA, which BPO uses for parameter-efficient SFT and for computing low-rank gradients for clustering.","marker":"Hu et al., 2021"}],"fun_headline_variants":["BPO matches DPO with just 10% of preference data","Balancing prompt vs response pairs cuts alignment data by 90%","Gradient-based clustering selects only 10% of alignment data","BPO balances breadth and depth to beat DPO with less data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that K-means clustering of projected LoRA gradients identifies the most informative response pairs, with the samples closest to a cluster centroid being the ones worth extra learning resources; the paper's evidence for this is the final benchmark scores, which were also used to pick the cluster counts and selection ratio.","fun_headline_variants_meta":{"raw":{"variants":["BPO matches DPO with just 10% of preference data","Balancing prompt vs response pairs cuts alignment data by 90%","Gradient-based clustering selects only 10% of alignment data","BPO balances breadth and depth to beat DPO with less data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000494,"raw_usage":{"total_tokens":2445,"prompt_tokens":987,"completion_tokens":1458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1384}},"tokens_in":603,"tokens_out":1458,"duration_ms":11945,"temperature":1.0,"reasoning_tokens":1384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:08:47.449371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the gradient-centroid selection in Eq. (6) with random selection of the same number of response pairs per prompt, keeping every other component fixed; if the random-selection run matches BPO's MT-Bench and AlpacaEval scores, then the claim that centroid proximity tracks informativeness is falsified. A second check is to train BPO on a held-out distribution (e.g., a different domain) and see whether the performance advantage over random depth allocation persists.","supporting_citations":[],"review_version":1}