Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that automatically extracted rules, scored by an LLM verifier and added to a learned reward during GRPO, improve preference alignment and reduce reward hacking.

desk verdict A genuinely new way to extract rule-based rewards from reasoning chains, but the headline numbers rest on a single seed and a manually added conciseness condition that undercuts the 'fully automated' claim. read the letter →

arxiv 2506.15651 v1 pith:KASNF2BD submitted 2025-06-18 cs.LG cs.AIcs.CL

Tevin Wang , Chenyan Xiong This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords AutoRulerule-basedrewardspreferencelearningRLHFGRPOrewardhackingLLM-as-a-judgereasoningchains
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AutoRule is a fully automated pipeline that reads pairwise preference data, asks a reasoning-capable LLM to explain why the chosen response is better, and converts those reasoning chains into a compact set of explicit rules. A smaller LLM-as-a-judge verifier then scores each candidate response by the fraction of rules it satisfies, and that rule score is added to the standard learned reward during GRPO training. The paper claims this hybrid reward improves preference alignment: on Llama-3-8B it reports a 28.6% relative gain in AlpacaEval 2.0 length-controlled win rate and a 6.1% relative gain in held-out MT-Bench second-turn performance over GRPO baselines using the same learned reward without the rule-based auxiliary. The paper also claims the rule-based reward is harder to game, showing less performance degradation over two training episodes than learned-reward baselines, and that the extracted rules are interpretable and dataset-specific.

What carries the argument

The load-bearing object is the AutoRule reward, defined as $r_{\mathrm{RA}}(x,y) = \frac{1}{K}\sum_{i=1}^K s_i$, where each $s_i \in \{0,1\}$ is a verifier's binary judgment of whether response $y$ satisfies rule $i$ for prompt $x$. The rule set is produced by a three-stage extractor: reasoning generation, per-chain rule extraction, and rule merging, which the paper reports compresses the candidate set to about 1--2% of its original size. The reward is rescaled ($\alpha=10$, $\beta=-7.5$), combined with the learned reward and KL penalty, and optimized with GRPO's group-relative advantage estimates. Its role is to give the policy a stable, interpretable signal that is not a learned continuous scalar, which the paper argues makes it less susceptible to reward hacking.

What would settle it

Train AutoRule for two episodes and, at every checkpoint, evaluate the verifier's rule-satisfaction judgments against held-out human preference labels on the policy's current outputs; if rule-reward scores rise while verifier-human agreement falls toward chance or the UltraFeedback/AlpacaEval win rates drop below the GRPO baseline, the auxiliary reward is being gamed rather than aligning the policy.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck for rule-based rewards in preference alignment is not the absence of good rules but the cost of writing them, and that reasoning chains make rule extraction automatic. Given a preference pair, a reasoning model is prompted to justify the preferred response, then to state the rules implicit in that justification; candidate rules are merged into one set; and a verifier model produces a binary pass/fail for each rule. The final reward is the mean rule satisfaction plus the learned reward model score minus a KL penalty, optimized with GRPO. On the paper's Llama-3-8B experiments, this yields the reported gains and, after two episodes, avoids the win-rate decline seen in learned-reward-only baselines, which the paper interprets as reduced reward hacking.

Load-bearing premise

The binary rule-satisfaction judgments made by the verifier must stay aligned with human preferences on the policy's newly generated responses during two episodes of optimization, even though those responses shift off the verifier's training distribution.

Editorial extensions

If this is right

  • Rule-based rewards for preference alignment no longer require manual rule engineering; any pairwise preference dataset can be converted into a rule set automatically.
  • Because the rules are explicit text, practitioners can inspect and audit the reward signal before training, something a learned reward model does not offer.
  • Adding the rule score to a learned reward raises both in-distribution win rate on UltraFeedback and out-of-distribution instruction-following on AlpacaEval 2.0 in the reported runs.
  • The rule reward remains effective over two training episodes, where the learned-reward-only GRPO baselines begin to decline, indicating less overoptimization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method's practical ceiling likely depends on verifier quality: a stronger or adversarially hardened verifier could enlarge the gains, while a cheaper verifier might be exploited as the policy shifts.
  • The extracted rule sets could be reused or combined across datasets, turning AutoRule into a way to build interpretable reward libraries for new alignment tasks.
  • The conciseness prompt found in the ablation suggests part of AutoRule's benefit is length control; a cleaner test would isolate the rule signal from the brevity prior by comparing against length penalties alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. AutoRule proposes a three-stage pipeline that extracts explicit preference rules from LLM reasoning chains over pairwise preference data, merges them into a compact rule set, and uses a Llama-3-8B verifier to produce a binary rule-satisfaction score. This rule score is combined with a learned reward model and a KL penalty inside GRPO during policy optimization. The paper reports improved UltraFeedback win rate, AlpacaEval 2.0 length-controlled win rate, and MT-Bench Turn 2 performance over PPO/GRPO baselines, and presents a two-episode reward-hacking analysis suggesting reduced overoptimization. The extracted rule sets are included in the appendix, and the code is open-sourced.

Significance. If the results hold, the paper makes a useful and timely contribution: it replaces manual rule engineering with an automatic extraction procedure, produces interpretable and dataset-specific rule sets, and provides an empirical link between rule-based auxiliary rewards and reduced reward hacking in preference learning. The paper is transparent about hyperparameters, releases code and extracted rules, and includes a case study that illustrates the advantage of reasoning-chain-based extraction. The main limitations are that all headline numbers come from a single training seed without variance estimates, and that the verifier's judgment quality under the policy's shifted response distribution is not measured; these gaps leave the central claims plausible but not yet firmly established.

major comments (5)
  1. [Section 5.2, Table 1; Appendix B, Table 7] All headline comparisons in Table 1 are based on a single training run at seed 42, with no confidence intervals or multiple seeds. For RL training, differences such as UltraFeedback win rate 77.2 vs 75.9 and MT-Bench Turn 2 7.83 vs 7.38 are within plausible run-to-run variation. Please report at least three seeds with mean and standard error for Table 1 and for the reward-hacking curves in Figures 3c and 3d, or explicitly qualify the claims as preliminary.
  2. [Section 5.1 and 5.3] The central mechanism is that the Llama-3-8B verifier's binary rule judgments remain aligned with human preferences on the policy's own generations, but this is not tested. The agreement and determinism analysis is performed on the original test distribution, while GRPO training shifts the response distribution over one or two episodes; if the verifier becomes uninformative or systematically outputs yes, the auxiliary reward could be a non-informative or harmful signal. Please measure per-rule agreement or calibration on policy-generated responses at intermediate checkpoints, using the checkpoints already collected for Figures 3c and 3d, and report the fraction of yes responses over training.
  3. [Section 4 and Table 2] The 'fully automated' claim is weakened by the manually added conciseness condition in the verifier prompt (Figure 11), which is not part of the extracted rules. Removing that condition drops AlpacaEval 2.0 LC WR from 21.6 to 16.5 and UltraFeedback WR from 77.2 to 74.6 (Table 2), so a hand-specified reward design choice is essential to the headline result. Please either justify the conciseness condition as part of the automated pipeline or revise the claims about full automation.
  4. [Section 5.1 and Appendix C] The rule agreement metric is computed only over response pairs in which a rule distinguishes between chosen and rejected responses, as stated in Section 5.1. Rules that rarely fire can therefore obtain high conditional agreement, and the reported percentages do not show how often each rule actually discriminates. Please also report the fraction of pairs on which each rule distinguishes and agreement computed over all pairs, or a precision/coverage analysis, so that the rule quality claims are not inflated.
  5. [Section 5.3, Figures 3c and 3d] The reward-hacking comparison uses one run per method, checkpoints every two steps, and evaluation on a 128-example subset, with no confidence intervals. The roughly five-point gap in Figure 3d could be within noise, and the conclusion that AutoRule reduces reward hacking for out-of-distribution data is not supported with variance estimates. Please provide bootstrap intervals over evaluation subsets or multiple-seed runs, or soften the claim to an exploratory observation.
minor comments (5)
  1. [Section 4] The MT-Bench human judgment dataset is cited as reference [15], but the correct reference is [26] (Zheng et al., Judging LLM-as-a-judge with MT-Bench and Chatbot Arena).
  2. [Table 2 and Section 5.4] The row labels in Table 2 are inconsistent with the definitions in Section 5.4: the text defines 'w/o Scaling' and 'w/o Scaling, Concise', while the table lists 'w/o Scaling, Concise' and 'w/o Concise'. Please clarify which variant corresponds to which row and fix the labels.
  3. [Section 3.2 and Section 4] The main method section defines rtotal using the unscaled rule reward rRA, while Section 4 introduces the scaled reward rRA' with alpha=10 and beta=-7.5. Please specify explicitly which reward is used in the final objective and how the scaling parameters were chosen.
  4. [Section 3.1] The notation 'R(i), r ~ pi_phi(·|x)' is ambiguous; the rule set should be written as a sample from the prompted distribution over rules, e.g., R(i) ~ pi_phi(·|prompt(r(i))).
  5. [Various] There are several typos: 'recieve' appears in Section 5.1, 'an reasoning chain-based' in Section 6, and 'a ablation' in Section 5.4. These should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the rule-extraction and auxiliary-reward pipeline is evaluated out-of-sample and does not reduce to its inputs by construction.

full rationale

AutoRule extracts rules from 256 UltraFeedback training examples and a 40-question MT-Bench training split using Deepseek-R1, then merges them into a fixed rule set. The rule reward is the mean of binary Llama-3-8B-Instruct verifier judgments over those rules, added to the same learned reward model used by the baselines. No equation in the paper defines the target evaluation metric (UltraFeedback win rate, AlpacaEval LC WR, MT-Bench turn score) in terms of the extracted rules or verifier outputs; the rules are generated once and then evaluated on held-out splits, with rule agreement measured against ground-truth preferences rather than optimized to match them. The scaling parameters alpha=10 and beta=-7.5 are chosen to align reward magnitudes and are ablated, not fit to the benchmark numbers. There are no self-citations, uniqueness theorems, or imported ansatze that carry the argument. The concern that the Llama-3-8B verifier's judgment quality under policy shift is unmeasured, or that single-seed runs lack error bars, is a robustness/correctness limitation, not a circularity: it does not make any reported result equivalent to its own input by construction. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The pipeline's assumptions are standard for LLM-as-a-judge reward models, but the verifier reliability and rule generalizability are not deeply tested.

free parameters (2)
  • rule reward scaling alpha = 10
    Hand-chosen to align rule-based reward magnitude with learned reward model for stable training (Section 3.2); the ablation in Section 5.4 shows a drop in performance without scaling.
  • rule reward offset beta = -7.5
    Hand-chosen alongside alpha to match the learned reward magnitude; also ablated in Section 5.4.
assumptions (3)
  • domain assumption LLM-as-a-judge verifier provides valid binary judgments of rule satisfaction
    Assumed throughout; the only evidence is rule agreement (~65-86%) and determinism at high temperature, not correctness under distribution shift.
  • domain assumption Rules extracted from 256 training examples generalize to the full preference distribution
    The extraction samples 256 UltraFeedback examples and the MT-Bench training split; the paper does not analyze rule coverage or stability across samples (Section 4).
  • domain assumption Mean of binary rule scores is a useful scalar reward for GRPO
    Section 3.2 defines rRA as the mean; no theoretical justification, only empirical results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning." pith.science (2026). https://pith.science/paper/KASNF2BD

@misc{pith2026250615651,
  author       = {Pith},
  title        = {Pith review of: AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KASNF2BD}},
  note         = {Machine review of arXiv:2506.15651}
}
read the original abstract

Rule-based rewards offer a promising strategy for improving reinforcement learning from human feedback (RLHF), but current approaches often rely on manual rule engineering. We present AutoRule, a fully automated method for extracting rules from preference feedback and formulating them into rule-based rewards. AutoRule extraction operates in three stages: it leverages a reasoning model to interpret user preferences, identifies candidate rules from the reasoning chain of these interpretations, and synthesizes them into a unified rule set. Leveraging the finalized rule set, we employ language-model verifiers to compute the fraction of rules satisfied by each output, using this metric as an auxiliary reward alongside the learned reward model during policy optimization. Training a Llama-3-8B model with AutoRule results in a 28.6\% relative improvement in length-controlled win rate on AlpacaEval2.0, and a 6.1\% relative gain in second-turn performance on a held-out MT-Bench subset, compared to a GRPO baseline trained with the same learned reward model but without the rule-based auxiliary reward. Our analysis confirms that the extracted rules exhibit good agreement with dataset preference. We find that AutoRule demonstrates reduced reward hacking compared to a learned reward model when run over two episodes. Finally, our case study suggests that the extracted rules capture unique qualities valued in different datasets. The extracted rules are provided in the appendix, and the code is open-sourced at https://github.com/cxcscmu/AutoRule.

Figures

Figures reproduced from arXiv: 2506.15651 by the authors.

Figure 1
Figure 1. Overview of the AUTORULE method. “disentangle” length from other features. Reward shaping methods such as PAR [8] and LSC [25] apply sigmoid or log-sigmoid transformations centered on reference model outputs or percentiles. Other approaches leverage multiple reward models: WARM [19] averages outputs from several reward models to reduce overoptimization, while ArmoRM [24] combines interpretable reward objectives usin… view at source ↗
Figure 2
Figure 2. Distributions of individual rule agreement in Figures 2a and 2b, rule score delta distributions [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Average individual rule scores of AUTORULE for two episodes, split into two figures (3a and 3b), evaluation of AUTORULE and GRPO baselines for two episodes in Figures 3c and 3d. the baseline, demonstrating the effectiveness of rule-based supervision for complex, multi-turn interactions. Beyond in-domain performance, our findings indicate that rule-based approaches exhibit superior generalization to out-of-distributi… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Justification per-rule [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Rule agreement matrix on UltraFeedback data [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Rule agreement matrix on MT-Bench Human Judgements data [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Case study: reasoning chain vs justification. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Justification (AUTORULE Extractor stage 1) prompt. Rule Extraction Prompt [Instruction] Based on the following reasoning about why conversation with assistant winner is better, extract any rule-like statements implied by the reasoning that indicate this preference. Rul…
Figure 9
Figure 9. Figure 9: Rule extraction (AUTORULE Extractor stage 2) prompt. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Rule merging (AUTORULE Extractor stage 3) prompt. Rule Verifier Prompt You are an impartial judge. Determine whether the AI assistant’s response in the following conversation both complies with the rule below and does so in a concise manner: Rule: {rule} [Start of Con…
Figure 11
Figure 11. Figure 11: Rule verifier prompt. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Rule verifier prompt (no conciseness). 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: UltraFeedback win-rate judgement prompt. [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Survey on Progress in LLM Alignment from the Perspective of Reward Design

    cs.CL 2025-05 conditional novelty 4.0 of 10

    This paper organizes the LLM alignment literature into a reward-design-centered taxonomy and claims the field's evolution runs from rule-based to learned rewards and from RL-based to RL-free optimization.

  2. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

28 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    The claude 3 model family: Opus, sonnet, haiku, 2024

    Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ Model_Card_Claude_3.pdf

    work page 2024

  2. [2]

    Training a helpful and harmless assistant with reinforcement learning from human feedback,

    Bai, Y ., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah,...

    work page

  3. [3]

    E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S

    Bai, Y ., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N.,...

    arXiv 2022

  4. [4]

    Odin: disentangled reward mitigates hacking in rlhf

    Chen, L., Zhu, C., Chen, J., Soselia, D., Zhou, T., Goldstein, T., Huang, H., Shoeybi, M., and Catanzaro, B. Odin: disentangled reward mitigates hacking in rlhf. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    work page 2024

  5. [5]

    Ultrafeedback: Boosting language models with high-quality feedback, 2024

    Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y ., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback, 2024. URL https://openreview. net/forum?id=pNkOx3IVWI

    work page 2024

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URLhttps://arxiv.org/abs/2501.12948

    arXiv 2025

  7. [7]

    Length-controlled alpacaeval: A simple debiasing of automatic evaluators

    Dubois, Y ., Liang, P., and Hashimoto, T. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. InFirst Conference on Language Modeling, 2024. URL https: //openreview.net/forum?id=CybBmzWBX0

    work page 2024

  8. [8]

    Reward shaping to mitigate reward hacking in rlhf, 2025

    Fu, J., Zhao, X., Yao, C., Wang, H., Han, Q., and Xiao, Y . Reward shaping to mitigate reward hacking in rlhf, 2025. URLhttps://arxiv.org/abs/2502.18770

    arXiv 2025

Show all 28 references
  1. [9]

    Scaling laws for reward model overoptimization

    Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.),Proceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Ma...

    2023

  2. [10]

    S., Green, R., Mokrá, S., Fernando, N., Wu, B., Foley, R., Young, S., Gabriel, I., Isaac, W., Mellor, J., Hassabis, D., Kavukcuoglu, K., Hendricks, L

    Glaese, A., McAleese, N., Tr˛ ebacz, M., Aslanides, J., Firoiu, V ., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., Campbell-Gillingham, L., Uesato, J., Huang, P.-S., Comanescu, R., Yang, F., See, A., Dathathri, S., Greig, R., Chen, C., Fritz, D., Elias, J. S....

    2022

  3. [11]

    Gemini: A family of highly capable multimodal models, 2025

    Google. Gemini: A family of highly capable multimodal models, 2025. URL https://arxiv. org/abs/2312.11805

    2025 arXiv

  4. [12]

    Openrlhf: An easy-to-use, scalable and high-performance rlhf framework, 2024

    Hu, J., Wu, X., Zhu, Z., Xianyu, Wang, W., Zhang, D., and Cao, Y . Openrlhf: An easy-to-use, scalable and high-performance rlhf framework, 2024. URL https://arxiv.org/abs/2405. 11143

    2024

  5. [13]

    The llama 3 herd of models, 2024

    Meta. The llama 3 herd of models, 2024. URLhttps://arxiv.org/abs/2407.21783. 10

    2024 arXiv

  6. [14]

    Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling

    Miao, Y ., Zhang, S., Ding, L., Bao, R., Zhang, L., and Tao, D. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.),Advances in Neu- ral Informat...

    2024

  7. [15]

    Rule based rewards for language model safety

    Mu, T., Helyar, A., Heidecke, J., Achiam, J., Vallone, A., Kivlichan, I., Lin, M., Beutel, A., Schulman, J., and Weng, L. Rule based rewards for language model safety. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.),Advances i...

    2024

  8. [16]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URLhttps://arxiv.org/abs/2303.08774

    2024 arXiv

  9. [17]

    F., Leike, J., and Lowe, R

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welin- der, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to...

    2022

  10. [18]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.),Advances in Neural Information Pr...

    2023

  11. [19]

    Warm: on the benefits of weight averaged reward models

    Ramé, A., Vieillard, N., Hussenot, L., Dadashi, R., Cideron, G., Bachem, O., and Ferret, J. Warm: on the benefits of weight averaged reward models. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  12. [20]

    Proximal policy optimiza- tion algorithms.CoRR, abs/1707.06347, 2017

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimiza- tion algorithms.CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347

    2017 arXiv

  13. [21]

    K., Wu, Y ., and Guo, D

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y . K., Wu, Y ., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URLhttps://arxiv.org/abs/2402.03300

    2024 arXiv

  14. [22]

    A long way to go: Investigating length correlations in RLHF, 2024

    Singhal, P., Goyal, T., Xu, J., and Durrett, G. A long way to go: Investigating length correlations in RLHF, 2024. URLhttps://openreview.net/forum?id=sNtDKdcI1f

    2024

  15. [23]

    M., Lowe, R., V oss, C., Radford, A., Amodei, D., and Christiano, P

    Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., V oss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize from human feedback, 2022. URL https: //arxiv.org/abs/2009.01325

    2022 arXiv

  16. [24]

    Interpretable preferences via multi- objective reward modeling and mixture-of-experts

    Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable preferences via multi- objective reward modeling and mixture-of-experts. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.),Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10582...

    2024 doi

  17. [25]

    Transforming and combining rewards for aligning large language models

    Wang, Z., Nagpal, C., Berant, J., Eisenstein, J., D’Amour, A., Koyejo, S., and Veitch, V . Transforming and combining rewards for aligning large language models. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  18. [26]

    E., and Stoica, I

    Zheng, L., Chiang, W.-L., Sheng, Y ., Zhuang, S., Wu, Z., Zhuang, Y ., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging 11 llm-as-a-judge with mt-bench and chatbot arena. In Oh, A., Naumann, T., Glober- son, A., Saenko, K., Hardt, M., and L...

  19. [2022]

    URLhttps://arxiv.org/abs/2204.05862

    arXiv

  20. [2023]

    confidence

    URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf. 12 A Discussion A.1 Limitations While our approach shows promising generalization from UltraFeedback to AlpacaEval 2.0, further work will be done...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.