{"id":"582d3ca5-b869-4dd4-9c3f-73f4a0326608","arxiv_id":"2506.15651","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AutoRule extracts rules from reasoning chains over preference data, scores responses against them with an LLM verifier, and uses the scores as an auxiliary reward, improving Llama-3-8B alignment metrics and showing reduced reward hacking.","lead":"AutoRule automatically turns preference judgments into a checklist of rules by reading a reasoning model's step-by-step explanations, then uses those rules as a second reward signal when training a language model. In tests, the method improved a Llama-3-8B model's win rates on AlpacaEval 2.0 and MT-Bench compared with standard GRPO training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gains hinge on a single seed, with no variance estimates; the rule verifier's judgment quality under policy shift is unmeasured, and the conciseness add-on breaks full automation.","rationale":"The reader's verdict is CONDITIONAL with medium correctness risk, and the reader identified the verifier drift under policy shift as the weakest assumption. I agree that this is a key unvalidated mechanism, but I would weight the absence of repeated runs and error bars more heavily because the headline improvements (28.6% relative on AlpacaEval, 6.1% on MT-Bench Turn 2, 1.7% on UltraFeedback WR) are small in absolute terms for the latter two and stem from single runs. The ablation also shows that the manual 'concise' condition in the verifier prompt (Figure 11) is essential: removing it drops AlpacaEval 2.0 LC WR from 21.6 to 16.5, and removing scaling drops it to 14.5, undercutting the claim that the automatically extracted rules alone drive the gain. The reward hacking experiment (Section 5.3) does show AutoRule maintains high win rates over two episodes while baselines decline, which is genuine independent support for the mechanism, and the rule agreement analysis (Section 5.1) is a reasonable static check. But neither directly measures verifier agreement on policy-generated responses during training, and no confidence intervals are provided anywhere. A conditional accept with a request for multi-seed runs and verifier calibration under distribution shift is the right call; I would not move to reject because the method is novel, reproducible infrastructure is provided, and the two-episode reward-hacking results are suggestive evidence that the rule reward is not actively harmful.","tokens_in":16987,"tokens_out":1721,"duration_ms":16310,"concrete_test":"Run the AutoRule and GRPO baseline condition three times with different seeds (e.g., 42, 7, 123) using identical hyperparameters, and report mean and standard deviation for UltraFeedback WR, AlpacaEval 2.0 LC WR, and MT-Bench Turn 2. If the 28.6% relative AlpacaEval improvement and the 6.1% MT-Bench Turn 2 gain are within one standard deviation of baseline performance, the central claim would lack statistical support.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim rests on a fragile empirical basis: every reported number comes from one training run at seed 42 (Appendix B), with no error bars or repeated runs. With ~1 percentage point differences on UltraFeedback win rate (77.2 vs 75.9) and MT-Bench Turn 2 (7.83 vs 7.38), single-run noise could change conclusions. More importantly, the claimed mechanism—that binary rule-satisfaction verifier judgments remain aligned with human preferences during two episodes of GRPO—is not tested. Section 5.1 measures verifier determinism (83.6%) and static agreement on the original test distribution, but Figures 3c/3d show only aggregate evaluation curves and reward hacking comparison; there is no measurement of per-rule agreement or calibration of the verifier on the policy's own generations after RL. If the Llama-3-8B verifier systematic mislabels shifted responses (e.g., always 'yes' once responses become polished, or changes with the policy's formatting), the auxiliary reward could be a non-informative or harmful signal, and the reported gain could come from the scaled reward or conciseness bias rather than from extracted rules. Additionally, the paper calls the method 'fully automated' yet Section 4 and the ablation (Table 2) show a manually added conciseness condition in the verifier prompt (Figure 11) is required for the headline result; removing it drops LC win rate from 21.6 to 16.5. Thus the central claim that automatic rule extraction is what drives improvements is not cleanly established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"AutoRule proposes a three-stage pipeline that extracts explicit preference rules from LLM reasoning chains over pairwise preference data, merges them into a compact rule set, and uses a Llama-3-8B verifier to produce a binary rule-satisfaction score. This rule score is combined with a learned reward model and a KL penalty inside GRPO during policy optimization. The paper reports improved UltraFeedback win rate, AlpacaEval 2.0 length-controlled win rate, and MT-Bench Turn 2 performance over PPO/GRPO baselines, and presents a two-episode reward-hacking analysis suggesting reduced overoptimization. The extracted rule sets are included in the appendix, and the code is open-sourced.","tokens_in":17398,"tokens_out":6483,"duration_ms":70233,"significance":"If the results hold, the paper makes a useful and timely contribution: it replaces manual rule engineering with an automatic extraction procedure, produces interpretable and dataset-specific rule sets, and provides an empirical link between rule-based auxiliary rewards and reduced reward hacking in preference learning. The paper is transparent about hyperparameters, releases code and extracted rules, and includes a case study that illustrates the advantage of reasoning-chain-based extraction. The main limitations are that all headline numbers come from a single training seed without variance estimates, and that the verifier's judgment quality under the policy's shifted response distribution is not measured; these gaps leave the central claims plausible but not yet firmly established.","major_comments":[{"comment":"All headline comparisons in Table 1 are based on a single training run at seed 42, with no confidence intervals or multiple seeds. For RL training, differences such as UltraFeedback win rate 77.2 vs 75.9 and MT-Bench Turn 2 7.83 vs 7.38 are within plausible run-to-run variation. Please report at least three seeds with mean and standard error for Table 1 and for the reward-hacking curves in Figures 3c and 3d, or explicitly qualify the claims as preliminary.","section":"Section 5.2, Table 1; Appendix B, Table 7"},{"comment":"The central mechanism is that the Llama-3-8B verifier's binary rule judgments remain aligned with human preferences on the policy's own generations, but this is not tested. The agreement and determinism analysis is performed on the original test distribution, while GRPO training shifts the response distribution over one or two episodes; if the verifier becomes uninformative or systematically outputs yes, the auxiliary reward could be a non-informative or harmful signal. Please measure per-rule agreement or calibration on policy-generated responses at intermediate checkpoints, using the checkpoints already collected for Figures 3c and 3d, and report the fraction of yes responses over training.","section":"Section 5.1 and 5.3"},{"comment":"The 'fully automated' claim is weakened by the manually added conciseness condition in the verifier prompt (Figure 11), which is not part of the extracted rules. Removing that condition drops AlpacaEval 2.0 LC WR from 21.6 to 16.5 and UltraFeedback WR from 77.2 to 74.6 (Table 2), so a hand-specified reward design choice is essential to the headline result. Please either justify the conciseness condition as part of the automated pipeline or revise the claims about full automation.","section":"Section 4 and Table 2"},{"comment":"The rule agreement metric is computed only over response pairs in which a rule distinguishes between chosen and rejected responses, as stated in Section 5.1. Rules that rarely fire can therefore obtain high conditional agreement, and the reported percentages do not show how often each rule actually discriminates. Please also report the fraction of pairs on which each rule distinguishes and agreement computed over all pairs, or a precision/coverage analysis, so that the rule quality claims are not inflated.","section":"Section 5.1 and Appendix C"},{"comment":"The reward-hacking comparison uses one run per method, checkpoints every two steps, and evaluation on a 128-example subset, with no confidence intervals. The roughly five-point gap in Figure 3d could be within noise, and the conclusion that AutoRule reduces reward hacking for out-of-distribution data is not supported with variance estimates. Please provide bootstrap intervals over evaluation subsets or multiple-seed runs, or soften the claim to an exploratory observation.","section":"Section 5.3, Figures 3c and 3d"}],"minor_comments":[{"comment":"The MT-Bench human judgment dataset is cited as reference [15], but the correct reference is [26] (Zheng et al., Judging LLM-as-a-judge with MT-Bench and Chatbot Arena).","section":"Section 4"},{"comment":"The row labels in Table 2 are inconsistent with the definitions in Section 5.4: the text defines 'w/o Scaling' and 'w/o Scaling, Concise', while the table lists 'w/o Scaling, Concise' and 'w/o Concise'. Please clarify which variant corresponds to which row and fix the labels.","section":"Table 2 and Section 5.4"},{"comment":"The main method section defines rtotal using the unscaled rule reward rRA, while Section 4 introduces the scaled reward rRA' with alpha=10 and beta=-7.5. Please specify explicitly which reward is used in the final objective and how the scaling parameters were chosen.","section":"Section 3.2 and Section 4"},{"comment":"The notation 'R(i), r ~ pi_phi(·|x)' is ambiguous; the rule set should be written as a sample from the prompted distribution over rules, e.g., R(i) ~ pi_phi(·|prompt(r(i))).","section":"Section 3.1"},{"comment":"There are several typos: 'recieve' appears in Section 5.1, 'an reasoning chain-based' in Section 6, and 'a ablation' in Section 5.4. These should be corrected.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The automatic rule extraction angle is timely and the released code and rules are valuable, but the single-seed empirical evidence and the absence of a verifier-shift analysis are current barriers. I would not reject the paper; with multiple-seed results and a direct test of verifier reliability under policy shift, the claims would be substantially stronger and the paper would be a good fit for the venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AutoRule is worth a look. The three-stage pipeline—reasoning-chain generation, rule extraction, rule merging—is a real departure from prior work that relies on human-written or crowd-sourced rules. The extracted rules are interpretable, the code and rule sets are open-sourced, and the reported gains on AlpacaEval 2.0 LC WR (21.6 vs. 16.8 for the best baseline) and MT-Bench Turn 2 (7.83 vs. 7.38) are meaningful if they hold. The rule agreement analysis is careful, and the determinism check on the verifier is a nice touch.\n\nThe soft spots are real but not disqualifying. Every headline number comes from one seed, with no error bars or repeated runs. Given that the UltraFeedback win-rate difference between AutoRule and GRPO is only about one point (77.2 vs. 75.9), single-run noise could change the ranking. The verifier's judgment quality under policy distribution shift is unmeasured; the paper reports static agreement on the test distribution, but not how well the binary rule judgments track human preferences on the policy's own generations after two episodes of GRPO. That is the load-bearing assumption, and it deserves direct testing.\n\nThe bigger concern is the conciseness condition. The verifier prompt in Figure 11 explicitly requires that the response \"complies with the rule below and does so in a concise manner.\" The ablation in Table 2 shows that removing this condition drops AlpacaEval LC WR from 21.6 to 16.5. So the headline result depends on a manually inserted bias, not just on the automatically extracted rules. That undercuts the \"fully automated\" framing and makes it hard to attribute the gain to rule extraction per se. The paper should at least discuss this openly, and ideally ablate the conciseness condition while keeping scaling fixed.\n\nThe reward hacking analysis is also thin: two episodes, checkpoints every two steps, and evaluation on only 128 examples. That is suggestive, not convincing.\n\nStill, the core idea is sound and the paper is honestly written. The limitations section is brief but the authors do acknowledge the need for broader evaluation. I would not desk-reject this. It deserves peer review, but the reviewers should press for multiple seeds, a verifier calibration study on shifted data, and a cleaner ablation that separates rule extraction from conciseness bias.","headline":"A genuinely new way to extract rule-based rewards from reasoning chains, but the headline numbers rest on a single seed and a manually added conciseness condition that undercuts the 'fully automated' claim.","tokens_in":17804,"tokens_out":1384,"would_cite":true,"duration_ms":17910,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that automatically extracted rules, scored by an LLM verifier and added to a learned reward during GRPO, improve preference alignment and reduce reward hacking.","keywords":["AutoRule","rule-based rewards","preference learning","RLHF","GRPO","reward hacking","LLM-as-a-judge","reasoning chains"],"falsifier":"Train AutoRule for two episodes and, at every checkpoint, evaluate the verifier's rule-satisfaction judgments against held-out human preference labels on the policy's current outputs; if rule-reward scores rise while verifier-human agreement falls toward chance or the UltraFeedback/AlpacaEval win rates drop below the GRPO baseline, the auxiliary reward is being gamed rather than aligning the policy.","tokens_in":16772,"feed_emoji":"🤖","tokens_out":6771,"duration_ms":70095,"temperature":0.7,"pith_summary":"AutoRule is a fully automated pipeline that reads pairwise preference data, asks a reasoning-capable LLM to explain why the chosen response is better, and converts those reasoning chains into a compact set of explicit rules. A smaller LLM-as-a-judge verifier then scores each candidate response by the fraction of rules it satisfies, and that rule score is added to the standard learned reward during GRPO training. The paper claims this hybrid reward improves preference alignment: on Llama-3-8B it reports a 28.6% relative gain in AlpacaEval 2.0 length-controlled win rate and a 6.1% relative gain in held-out MT-Bench second-turn performance over GRPO baselines using the same learned reward without the rule-based auxiliary. The paper also claims the rule-based reward is harder to game, showing less performance degradation over two training episodes than learned-reward baselines, and that the extracted rules are interpretable and dataset-specific.","feed_headline":"Auto rules beat learned-reward-only RLHF by 28.6 percent","feed_subtitle":"A three-stage pipeline mines preference reasoning chains into verifiable rule rewards that also resist reward hacking.","key_machinery":"The load-bearing object is the AutoRule reward, defined as $r_{\\mathrm{RA}}(x,y) = \\frac{1}{K}\\sum_{i=1}^K s_i$, where each $s_i \\in \\{0,1\\}$ is a verifier's binary judgment of whether response $y$ satisfies rule $i$ for prompt $x$. The rule set is produced by a three-stage extractor: reasoning generation, per-chain rule extraction, and rule merging, which the paper reports compresses the candidate set to about 1--2% of its original size. The reward is rescaled ($\\alpha=10$, $\\beta=-7.5$), combined with the learned reward and KL penalty, and optimized with GRPO's group-relative advantage estimates. Its role is to give the policy a stable, interpretable signal that is not a learned continuous scalar, which the paper argues makes it less susceptible to reward hacking.","core_discovery":"The central claim is that the bottleneck for rule-based rewards in preference alignment is not the absence of good rules but the cost of writing them, and that reasoning chains make rule extraction automatic. Given a preference pair, a reasoning model is prompted to justify the preferred response, then to state the rules implicit in that justification; candidate rules are merged into one set; and a verifier model produces a binary pass/fail for each rule. The final reward is the mean rule satisfaction plus the learned reward model score minus a KL penalty, optimized with GRPO. On the paper's Llama-3-8B experiments, this yields the reported gains and, after two episodes, avoids the win-rate decline seen in learned-reward-only baselines, which the paper interprets as reduced reward hacking.","pith_inferences":["The method's practical ceiling likely depends on verifier quality: a stronger or adversarially hardened verifier could enlarge the gains, while a cheaper verifier might be exploited as the policy shifts.","The extracted rule sets could be reused or combined across datasets, turning AutoRule into a way to build interpretable reward libraries for new alignment tasks.","The conciseness prompt found in the ablation suggests part of AutoRule's benefit is length control; a cleaner test would isolate the rule signal from the brevity prior by comparing against length penalties alone."],"forward_implications":["Rule-based rewards for preference alignment no longer require manual rule engineering; any pairwise preference dataset can be converted into a rule set automatically.","Because the rules are explicit text, practitioners can inspect and audit the reward signal before training, something a learned reward model does not offer.","Adding the rule score to a learned reward raises both in-distribution win rate on UltraFeedback and out-of-distribution instruction-following on AlpacaEval 2.0 in the reported runs.","The rule reward remains effective over two training episodes, where the learned-reward-only GRPO baselines begin to decline, indicating less overoptimization."],"supporting_citations":[{"why":"Motivates rule-based rewards as a reward-hacking countermeasure and supplies the reasoning model used for extraction.","marker":"[6]"},{"why":"Provides the GRPO algorithm and group-relative advantage estimator used in the RL stage.","marker":"[21]"},{"why":"Establishes the LLM-as-a-judge paradigm and the MT-Bench evaluation used for verifier judgments and held-out multi-turn testing.","marker":"[26]"},{"why":"Supplies the UltraFeedback preference dataset that AutoRule extracts rules from and trains on.","marker":"[5]"},{"why":"Provides the Llama-3-8B base model, SFT checkpoint, and verifier used in the experiments.","marker":"[13]"},{"why":"Defines AlpacaEval 2.0 and its length-controlled win rate, the main out-of-distribution evaluation.","marker":"[7]"},{"why":"Frames the RLHF pipeline that AutoRule modifies with rule-based auxiliary rewards.","marker":"[17]"},{"why":"Prior work on decomposing policies into rule-based rewards for safety, which AutoRule extends to automated preference alignment.","marker":"[15]"}],"fun_headline_variants":["AutoRule: reasoning chains auto-extract reward rules","28.6% win-rate gain from auto-mined rule rewards","AutoRule reduces reward hacking via rule rewards","Automated rule extraction beats learned-reward RLHF","AutoRule: from preference reasoning to verifiable rules"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The binary rule-satisfaction judgments made by the verifier must stay aligned with human preferences on the policy's newly generated responses during two episodes of optimization, even though those responses shift off the verifier's training distribution.","fun_headline_variants_meta":{"raw":{"variants":["AutoRule: reasoning chains auto-extract reward rules","28.6% win-rate gain from auto-mined rule rewards","AutoRule reduces reward hacking via rule rewards","Automated rule extraction beats learned-reward RLHF","AutoRule: from preference reasoning to verifiable rules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000481,"raw_usage":{"total_tokens":2389,"prompt_tokens":965,"completion_tokens":1424,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1347}},"tokens_in":581,"tokens_out":1424,"duration_ms":14845,"temperature":1.0,"reasoning_tokens":1347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:51:28.670741+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train AutoRule for two episodes and, at every checkpoint, evaluate the verifier's rule-satisfaction judgments against held-out human preference labels on the policy's current outputs; if rule-reward scores rise while verifier-human agreement falls toward chance or the UltraFeedback/AlpacaEval win rates drop below the GRPO baseline, the auxiliary reward is being gamed rather than aligning the policy.","supporting_citations":[{"cited_title":"E., and Stoica, I","cited_arxiv_id":null,"evidence_quote":"Establishes the LLM-as-a-judge paradigm and the MT-Bench evaluation used for verifier judgments and held-out multi-turn testing."},{"cited_title":"Ultrafeedback: Boosting language models with high-quality feedback, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the UltraFeedback preference dataset that AutoRule extracts rules from and trains on."},{"cited_title":"Length-controlled alpacaeval: A simple debiasing of automatic evaluators","cited_arxiv_id":null,"evidence_quote":"Defines AlpacaEval 2.0 and its length-controlled win rate, the main out-of-distribution evaluation."},{"cited_title":"F., Leike, J., and Lowe, R","cited_arxiv_id":null,"evidence_quote":"Frames the RLHF pipeline that AutoRule modifies with rule-based auxiliary rewards."},{"cited_title":"Rule based rewards for language model safety","cited_arxiv_id":null,"evidence_quote":"Prior work on decomposing policies into rule-based rewards for safety, which AutoRule extends to automated preference alignment."}],"review_version":1}