{"id":"f5574512-684e-461f-ba06-961fb996084e","arxiv_id":"2501.15453","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Selecting the five safety rules with largest response discrepancy maximizes mutual information under stated assumptions, and a reward model trained with this adaptive labeling achieves top RewardBench safety.","lead":"This paper proposes a way to automatically pick which safety rules to apply when labeling preference data for reward model training: for each pair of responses, choose the five rules where the responses differ most. An 8B reward model trained on 1K such labels reaches the top of the RewardBench safety leaderboard as of January 25, 2025.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.4's proof relies on a false additivity identity for mutual information under conditional independence; the claimed optimality of max-discrepancy selection is therefore unproven as stated.","rationale":"The paper's central claim has two parts: a theoretical guarantee (Theorem 3.4) and an empirical result (RAMO scoring 95.1 on RewardBench-Safety). I read the theorem as the load-bearing part because the abstract and contributions explicitly advertise a proof that max-discrepancy selection maximizes mutual information with hidden ground-truth labels. The proof's additivity step is mathematically incorrect: conditional independence given H does not imply additive mutual information with H. This is not merely an unvalidated modeling assumption; it is an internally false step, and it is more fundamental than the reader's identified gap between Eq. (11) and Eq. (12). The two-rule computation with d1=d2=2 demonstrates the failure of the asserted identity. I do not claim the theorem's conclusion is necessarily false; it may be salvageable by a different argument, which is why the concrete test enumerates small cases to check the conclusion. The empirical ablation and leaderboard result are genuine evidence, and the paper's main empirical method is clearly described. However, the current manuscript cannot be accepted with the advertised theoretical guarantee intact. Since the reader's verdict was already CONDITIONAL, my concern sharpens the required revisions but does not move the category; I would keep the verdict CONDITIONAL and require the authors to either provide a correct proof of Theorem 3.4 or explicitly downgrade the theoretical claim, while also addressing the reproducibility issues already noted.","tokens_in":23894,"tokens_out":36892,"duration_ms":309707,"concrete_test":"Recompute I(T_s;H) for the two-rule case d1=d2=2 under the exact assumptions of Appendix A: the proof's asserted equality gives 0.862 bits, while the exact joint computation gives ≈0.686 bits, settling that the additivity step is false. To further check whether the theorem's conclusion survives despite the invalid proof, exhaustively enumerate all subsets of size r=2 for R=3 with, e.g., d=(2,2,1) and d=(5,5,4.9), and compare the subset maximizing Σ|d_i| with the subset maximizing I(T_s;H); if any winning MI subset has a lower Σ|d_i|, Theorem 3.4's conclusion is false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Appendix A, the proof asserts: 'Under conditional independence, the mutual information of I(T_s; H) is a sum of individual mutual information: I(T_s; H)=Σ_{i∈Is} I(T_i; H).' This identity is false. Conditional independence given H only makes conditional entropies additive, H(T_s|H)=Σ_i H(T_i|H); it does not make marginal entropies additive. Hence I(T_s;H)=H(T_s)-Σ_i H(T_i|H) is not Σ_i [H(T_i)-H(T_i|H)]. A direct computation under the paper's own model (H uniform; T_i independent given H with P(T_i=+1|H=+1)=σ(d_i), P(T_i=-1|H=-1)=σ(d_i)) for d1=d2=2 gives I(T_1;H)=I(T_2;H)=0.431 bits, but I(T_1,T_2;H)≈0.686 bits, not 0.862 bits. Thus the central step linking max discrepancy to max mutual information is invalid, and Theorem 3.4 is not proven. The theorem also does not cover the relevance term γ·sim(x,u_i) in Eq. (12), so even a corrected proof would not directly justify the implemented method. The empirical results may stand, but the theoretical guarantee advertised in the abstract is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a data-adaptive rule-selection method for reward-model training. Given a large pool of safety rules, it selects, for each trio of prompt and two responses, the r rules with the largest rating discrepancy between the two responses, augmented by a prompt-relevance regularization term. A 3B Rule Adapter is trained to imitate this selection on 63K pairwise comparisons, and the resulting 5 selected rules are used to label 1K preference trios, from which an 8B reward model (RAMO) is trained. The paper reports a RewardBench safety score of 95.1, the highest on the leaderboard as of January 25, 2025, along with ablations against random, fixed, all-rule, GPT-selected, and direct-GPT-labeling baselines, a generalization experiment on HH-RLHF, and a PPO alignment experiment. The central theoretical claim is Theorem 3.4, stating that max-discrepancy selection maximizes the mutual information between rule-based preference labels and a hidden ground-truth label.","tokens_in":24191,"tokens_out":5640,"duration_ms":50460,"significance":"If the theoretical result and the empirical results hold, the paper makes a useful contribution: it offers an interpretable, low-cost alternative to full rule enumeration for fine-grained reward-model annotation, and it releases the rule pool, synthetic dataset, Rule Adapter, and RAMO. The empirical ablation is informative: RAMO beats dynamic-random, fixed, all-rule, and GPT-selected baselines, and the HH-RLHF relabeling experiment shows consistent gains over human labels in the paper's settings. The main reservation is that the advertised theoretical guarantee is central to the paper's framing, and the proof of Theorem 3.4 contains a load-bearing error. The empirical results may stand independently, but the paper's claim to have 'demonstrated theoretically' that max-discrepancy selection maximizes mutual information is not currently supported.","major_comments":[{"comment":"The proof asserts that under conditional independence of the rule votes T_i given H, the mutual information decomposes as I(T_s; H) = sum_{i in I_s} I(T_i; H). This identity is false. Conditional independence given H makes conditional entropies additive, H(T_s | H) = sum_i H(T_i | H), but it does not make marginal entropies additive, so I(T_s; H) = H(T_s) - H(T_s | H) is not generally equal to sum_i [H(T_i) - H(T_i | H)]. Under the paper's own model, taking H uniform and T_i independent given H with P(T_i = +1 | H = +1) = P(T_i = -1 | H = -1) = sigma(d_i), setting d_1 = d_2 = 2 gives I(T_1; H) = I(T_2; H) ≈ 0.473 bits, while I(T_1, T_2; H) ≈ 0.687 bits rather than the claimed 0.946 bits. Therefore the proof's reduction of the joint mutual information to a sum of per-rule Jensen-Shannon terms is invalid, and Theorem 3.4 is not established as stated.","section":"Appendix A, proof of Theorem 3.4"},{"comment":"The proof introduces two modeling assumptions that do not appear in the theorem statement and are not validated against any preference data: (i) the rule labels T_i are conditionally independent given the hidden label H, and (ii) each rule's vote satisfies P(T_i = +1 | H = +1) = P(T_i = -1 | H = -1) = sigma(psi_i(v_A) - psi_i(v_B)). These are substantive behavioral assumptions about how rule-based raters relate to the hidden ground truth. The theorem as stated claims a general equivalence between max-discrepancy selection and maximum mutual information; if the theorem is to be retained, these conditions must be stated as hypotheses, and the paper should provide evidence that the sigmoid conditional-independence model is a reasonable description of the actual rule-rating process.","section":"Appendix A, assumptions of Theorem 3.4"},{"comment":"The actual selection method used in the experiments, including the Rule Adapter training and the reward-model data labeling, is the regularized objective of Eq. (12), which contains the term gamma * sim(x, u_i) with gamma = 2 chosen in Appendix G.1.2. Theorem 3.4, however, addresses only the unregularized objective of Eq. (11). Even if the additivity problem in Appendix A were repaired, the theorem would not justify the exact method evaluated. The paper should either extend the theoretical analysis to cover the relevance regularization or explicitly restrict the theoretical claim to the unregularized surrogate.","section":"Section 3.3 (Eq. 12) and Section 3.4 (Theorem 3.4, Eq. 15)"}],"minor_comments":[{"comment":"The displayed definition of conditional entropy H(U | V) is missing the minus sign: it should be H(U | V) = -E_{(u,v) ~ P(U,V)} log P_{U|V}(u | v). The subsequent entropy calculations use the correct sign, but the displayed definition is mathematically wrong.","section":"Appendix A, Definition A.1"},{"comment":"The proof contains internal citation inconsistencies: it refers to 'Lemma A.4' and 'Lemma A.5', but the relevant lemmas are named Lemma A.5 and Lemma A.6. In addition, the additivity equation writes I(T_s; H) = sum_{i in I_s} I(T_i; Y), using Y where the theorem uses H.","section":"Appendix A, proof of Theorem 3.4"},{"comment":"Equation (15) writes psi(v_B) without the rule index; it should be psi_i(v_B), matching the discrepancy term |psi_i(v_A) - psi_i(v_B)| used in Eq. (11).","section":"Theorem 3.4, Eq. (15)"},{"comment":"Definition 3.1 defines psi_i as taking values in [0, 1], but the actual implementation described in Appendix D uses the score P(Yes) - P(No) with range [-1, 1]. The authors note that shifting and scaling does not affect comparisons, but the notation should be made consistent for clarity.","section":"Appendix D"},{"comment":"Table 9's caption reads 'xxxx', which appears to be an unfinished placeholder. The hyperparameter table should be given a proper caption and, ideally, a description of the number of repeated runs.","section":"Appendix G.1.4, Table 9"},{"comment":"The RAMO row reports a single set of scores with no indication of whether these are from a single run or averaged over seeds. Since Table 2 reports three-trial averages and Table 3 reports two-seed averages with standard deviations, the paper should state the number of runs for the headline Table 1 result.","section":"Section 5, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuinely interesting empirical pipeline and the release of resources is valuable, but the theoretical contribution advertised in the abstract and in Theorem 3.4 is currently unsupported: the additivity identity in Appendix A is false, and the theorem does not cover the regularized objective actually used. These are load-bearing issues for the paper's central claim, not merely presentation problems. I would advise the editor that the manuscript is not acceptable in its current form; a revision that either supplies a correct proof under explicitly stated, validated assumptions or substantially weakens the theoretical claim could make it acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one for the empirical idea, not for the theory. The core suggestion—select a small set of safety rules per response pair by maximum rating discrepancy, using a trained 3B Rule Adapter, then label 1K trios and train an 8B reward model—is genuinely new relative to fixed-rule multi-attribute reward models and Constitutional AI's random constitutions, and the ablation shows it works: RAMO beats random 5 rules, fixed 5 rules, all 100 rules, GPT-4-selected rules, and direct GPT preference labeling on RewardBench-Safety. That alone justifies a serious look.\n\nWhat it does well: the rule pool construction via DPP is sensible; the use of logits with an 'Irrelevant' third choice avoids forcing judgments on irrelevant rules; the HH-RLHF relabeling experiment includes error bars and actually shows the method outperforming human labels. The RLHF extension is a nice bonus.\n\nThe soft spots are significant. Theorem 3.4, the advertised theoretical guarantee, is not proven. The appendix's key step says that under conditional independence, I(T_s; H) = sum of individual I(T_i; H). That identity is false: conditional independence given H only makes conditional entropies additive. Check directly under the paper's own model: for d1 = d2 = 2, each rule alone has I = 0.431 bits, but the pair has 0.686 bits, not 0.862. So the claimed equivalence between max discrepancy and max mutual information fails as stated. The theorem also ignores the relevance term gamma * sim(x, u_i) in Eq. (12) that the experiments actually use. This is not a minor gap; the abstract and contribution list rest on this theorem. The authors should either supply a correct proof or, more honestly, reframe the criterion as a heuristic supported by the ablation.\n\nReproducibility is also short: the release claims lack URLs and hashes, the headline Table 1 has no error bars (the ablation table says 'averaged over 3 trials' but gives no variance), and there's an unfinished 'Table 9: xxxx' in G.1.4. Fixable, but sloppy.\n\nBottom line: the empirical method and ablation are worth referee time, and the paper would likely be a useful resource once the theory is corrected or withdrawn. Send it to peer review, but with major revisions required. If you work on cheap synthetic preference data or safety reward models, cite the empirical result, not the theorem.","headline":"The adaptive rule-selection idea and ablation are genuinely useful, but the paper's theoretical guarantee (Theorem 3.4) is invalid as proven, so it needs major revisions before its claims can be trusted.","tokens_in":24733,"tokens_out":6467,"would_cite":true,"duration_ms":51352,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that labeling preference data with the few rules on which the two responses differ most is provably the most informative choice, and an 8B reward model built this way tops the safety leaderboard.","keywords":["reward models","safety alignment","rule-based annotation","max-discrepancy selection","mutual information","RLHF","preference learning","data-adaptive labeling"],"falsifier":"Take a preference dataset whose true labels are known, compute the joint mutual information $I(T_s; H)$ for all $r$-subsets of rules, and check whether the max-discrepancy subset achieves the maximum; a counterexample would settle the claim. A simpler check is to measure whether rule votes remain independent conditional on the hidden truth, since substantial residual correlation would invalidate the additivity step.","tokens_in":23665,"feed_emoji":"🛡️","tokens_out":7571,"duration_ms":66270,"temperature":0.7,"pith_summary":"This paper tries to establish that, when reward models are trained from fine-grained rule-based safety annotations, the best rules to apply are the ones on which the two candidate responses differ most. The authors prove that max-discrepancy rule selection maximizes the mutual information between the resulting rule-based preference label and the hidden ground-truth preference, assuming each rule's vote is conditionally independent given that truth. They train a small Rule Adapter to pick the five most discrepant and relevant rules for each trio, use an LLM judge to rate responses on those rules, and train an 8B reward model, RAMO, on just 1K labeled trios. On the RewardBench safety benchmark this model scored 95.1, the top safety score as of January 25, 2025, and using it as the reward model in PPO also improved policy safety. The claim matters because it offers an automatic, interpretable way to label preference data for safety alignment while making far fewer annotation calls than scoring every rule.","feed_headline":"Safety score 95.1 with five most divisive safety rules","feed_subtitle":"An 8B safety reward model trained on only 1K adaptively labeled trios tops the RewardBench safety leaderboard.","key_machinery":"The Rule Adapter is a multi-label classifier built on a small instruction-tuned language model; it maps a prompt and two responses to the five rules with the largest rating discrepancy, regularized by cosine relevance between each rule and the prompt. The mathematical engine is the identity $I(T; H) = D_{\\mathrm{JS}}(P^+ \\parallel P^-)$, combined with the fact that the Jensen-Shannon divergence between $\\mathrm{Bern}(\\sigma(d))$ and $\\mathrm{Bern}(\\sigma(-d))$ increases strictly with $|d|$. Under conditional independence of rule votes, this makes selecting the largest-$|d_i|$ rules equivalent to maximizing $I(T_s; H)$, and the reward model RAMO is then trained with the Bradley-Terry loss on the resulting binary preferences.","core_discovery":"On the paper's own terms, the central discovery is a selection-to-information equivalence: given a budget of $r$ rules, the $r$ rules with the largest absolute score differences between the two responses are exactly the rules that maximize the mutual information $I(T_s; H)$ between the aggregated rule-based preference label and the hidden ground-truth label $H$. The proof routes through Jensen-Shannon divergence: each rule's conditional vote distribution is modeled as $P_i^+ \\sim \\mathrm{Bern}(\\sigma(d_i))$ and $P_i^- \\sim \\mathrm{Bern}(\\sigma(-d_i))$, whose divergence is $\\log 2 - H(\\sigma(d_i))$, strictly increasing in $|d_i|$, so under conditional independence the additive mutual information is maximized by picking the largest $|\\psi_i(v_A) - \\psi_i(v_B)|$. In practice the selection adds a relevance term $\\gamma \\cdot \\mathrm{sim}(x, u_i)$ and is distilled into a 3B Rule Adapter that outputs five rules per trio; the resulting 8B reward model attains 95.1 on RewardBench-Safety.","pith_inferences":["The paper leaves implicit that the same selection rule should transfer to non-safety multi-criteria settings such as helpfulness, factuality, or reasoning, but the conditional-independence assumption may need checking because those criteria often overlap more than safety rules do.","A testable extension the paper does not develop is a relevance-regularized version of the theorem: the proof covers the unregularized objective, while the deployed system optimizes the regularized objective, so the guarantee is not directly about the exact method used in experiments.","Because the label chain begins with an LLM judge's ratings, the method inherits any judge bias; one could turn the Rule Adapter into an audit tool that surfaces which rules drive each preference judgment, an application the paper does not explore."],"forward_implications":["If the equivalence holds, any reward-model training pipeline can cut its annotation budget to a handful of adaptively chosen rules without losing information about the true preference.","Rule-based labels produced this way can replace or refine human labels: in the paper's experiments, reward models trained on a human-preference dataset relabeled by the Rule Adapter outperformed models trained on the original human labels.","Because the Rule Adapter is small and trained once, the labeling step becomes cheaper and more interpretable than repeatedly querying a large judge model.","The PPO results imply that safety-critical RLHF fine-tuning can use a data-adaptively labeled reward model to improve policy safety from a small prompt budget."],"supporting_citations":[{"why":"Supplies the Bradley-Terry model used for reward-model training and for modeling each rule's conditional vote probability in the proof.","marker":"Bradley & Terry (1952)"},{"why":"Defines the KL divergence underlying the Jensen-Shannon divergence used to connect mutual information to rule discrepancy.","marker":"Kullback & Leibler (1951)"},{"why":"Defines RewardBench-Safety, the evaluation benchmark on which RAMO attains its safety score.","marker":"Lambert et al. (2024)"},{"why":"Hosts the RewardBench leaderboard whose comparison scores are reported in Table 1.","marker":"Allen Institute for AI (2024)"},{"why":"Provides the random-rule Constitutional AI baseline that the dynamic selection method is compared against.","marker":"Bai et al. (2022b)"},{"why":"Supplies the large constitution-style rule pool and rule-generation reference used to build the 100-rule pool.","marker":"Huang et al. (2024)"},{"why":"Contributes the PKU safety categories that guide rule generation and the Alpaca-7B response generator used for synthetic data.","marker":"Ji et al. (2024)"},{"why":"Provides the Skywork backbone whose weights initialize the RAMO reward model.","marker":"Liu et al. (2024)"},{"why":"Names the Llama3-70B-Instruct judge used for logits-based rule rating of responses.","marker":"Meta AI (2024b)"}],"fun_headline_variants":["Adaptive rule selection tops safety reward benchmark","Max-info rule pick boosts safety to 95.1","8B reward model hits 95.1 with adaptive rules","Divisive rules key to safety reward model peak","Mutual-information rule selection leads safety"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on assuming that, once you know which answer is truly better, each rule's vote is independent of every other rule's vote; if real rules vote in correlated ways, the proof's key additivity step does not go through, and the theorem also does not cover the relevance-weighted selection actually used in the experiments.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive rule selection tops safety reward benchmark","Max-info rule pick boosts safety to 95.1","8B reward model hits 95.1 with adaptive rules","Divisive rules key to safety reward model peak","Mutual-information rule selection leads safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1471,"prompt_tokens":965,"completion_tokens":506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":431}},"tokens_in":581,"tokens_out":506,"duration_ms":4908,"temperature":1.0,"reasoning_tokens":431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:16:24.699114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a preference dataset whose true labels are known, compute the joint mutual information $I(T_s; H)$ for all $r$-subsets of rules, and check whether the max-discrepancy subset achieves the maximum; a counterexample would settle the claim. A simpler check is to measure whether rule votes remain independent conditional on the hidden truth, since substantial residual correlation would invalidate the additivity step.","supporting_citations":[{"cited_title":"Collective Constitutional AI: Aligning a language model with public input","cited_arxiv_id":null,"evidence_quote":"Supplies the large constitution-style rule pool and rule-generation reference used to build the 100-rule pool."}],"review_version":1}