Pith. sign in

REVIEW 3 major objections 4 minor 85 references

A Systematic Analysis of Base Model Choice for Reward Modeling

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims the base model is a critical, overlooked hyperparameter for reward modeling: swapping Llama-3.x for the best similar-size model improves RewardBench by up to 14%, and a five-benchmark regression predicts most of that…

desk verdict A useful empirical sweep of 40 base models for reward modeling, but the headline model-selection claim is in-sample and needs re-validation before being cited as predictive. read the letter →

arxiv 2505.10775 v1 pith:WBGO6T5N submitted 2025-05-16 cs.CL cs.AI

Kian Ahrabian , Pegah Jandaghi , Negar Mokhberian , Sai Praneeth Karimireddy , Jay Pujara This is my paper · ORCID
classification cs.CLcs.AI
keywords rewardmodelingbasemodelselectionBenchHelpSteer2benchmarkcorrelationpost-trainingpre-trainingdatadistribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reward models are the scoring components used in reinforcement learning from human feedback; they rate which of two responses a user would prefer. This paper argues that the base model those scores are built on is a decisive hyperparameter, not a fixed default. Training the same reward-modeling objective on 40 chat models grouped by parameter count, it reports that replacing the common Llama-3.x base with the best model of similar size improves RewardBench by 3 to 14 percent, with Qwen2.5 and Gemma-2 models consistently ahead. The paper also claims that reward-model quality can be predicted before training: a simple linear combination of five reported benchmark scores beats any single benchmark at picking the top models, raising average top-5/10 overlap by about 18 percent. Its post-training analysis finds supervised fine-tuning is the stage that matters most (+15.5%), while later alignment steps cost 3 to 5 percent; the upshot is a practical recipe — treat the base model as a search axis, and use cheap benchmark proxies to choose it.

What carries the argument

The load-bearing setup is a controlled model-selection protocol. Forty chat models are partitioned into three parameter buckets ($<3$B, $3$–$6$B, $\geq6$B); each is fine-tuned on the same HelpSteer2(-Preference) data under both a Bradley-Terry pairwise loss and a multi-attribute regression loss, and the best checkpoint per model is scored on RewardBench. Selection quality is quantified by top-$k$ coverage, $C(\beta,\rho,L,k)=|T_\beta(L,k)\cap T_\rho(L,k)|/k$, the fraction of the true top-$k$ reward models that a benchmark also ranks in its top-$k$. Prediction is carried by a 10-fold Elastic Net over benchmark scores and parameter count. The pre-training probe is the length-normalized token log-probability $S_\phi(D,N)=\frac{1}{N}\sum_{i=1}^{N}\log p_\phi(t_i|t_{1:i-1})$ computed on a 1M-document SlimPajama sample. These pieces let the paper turn a costly search over base models into a cheap regression problem while keeping training conditions fixed.

What would settle it

Retrain a subset of the 40 models across several random seeds and on at least one additional preference dataset, such as UltraFeedback. If the same-size ranking changes so that a different family leads, or the gap between Llama-3.x and the best alternative falls below the seed-to-seed spread, the headline 3–14% gain cannot be attributed cleanly to base-model choice.

Watch

Extended reading notes

Core claim

The central discovery is an empirical regularity: holding the reward-modeling dataset, objective, and training budget fixed, the choice of base chat model changes RewardBench accuracy by up to 14 percent relative to the Llama-3.x default, within each of three parameter buckets (<3B, 3–6B, ≥6B). The paper further establishes that strong benchmark correlations (Pearson and Spearman, many at 0.8 or above) are misleading for model selection because top-rank coverage is low; a 10-fold Elastic Net trained on a small set of reported scores — two coding benchmarks, one safety benchmark, one instruction-following benchmark, and parameter count — recovers much of the ranking and improves top-5/10 overlap by about 18 percent on average. In post-training, supervised fine-tuning contributes the largest share of reward-modeling ability, and DPO/RLVR alignment stages reverse part of that gain. Finally, estimated pre-training data composition differs measurably across model families and, added to the same regression, lowers mean absolute prediction error from 3.2% to 1.7%.

Load-bearing premise

The paper treats one training run per base model, with a small learning-rate grid and best-checkpoint selection on the HelpSteer2 validation set, as a stable measure of each base model's reward-modeling quality; if rerunning the same model with different seeds or hyperparameters changes the score by as much as the reported gaps, the ranking is not reliable.

Editorial extensions

If this is right

  • Practitioners can gain 3–14% RewardBench score by switching the base model within the same size class, with no changes to training data or objective.
  • Qwen2.5 and Gemma-2 families are reliable alternatives to Llama-3.x as reward-model bases at the sizes tested, so defaulting to one family is a measurable opportunity cost.
  • A linear model over five reported numbers — two coding benchmarks, a safety benchmark, an instruction-following benchmark, and parameter count — can serve as a cheaper model-selection screen than training every candidate.
  • Supervised fine-tuning is the load-bearing post-training stage for reward modeling, and running later alignment stages such as DPO or RLVR can reduce the final RewardBench score by 3–5%.
  • Estimated pre-training data composition carries usable signal for predicting reward-model quality, reducing mean absolute prediction error from 3.2% to 1.7% when added to the benchmark regression.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to run the same selection recipe on a different reward training set, such as UltraFeedback; if the five-feature regression still ranks winners, the result transfers beyond HelpSteer2.
  • The paper's data-composition probe could be turned into a family-level correction term for reward-model scaling laws, something the authors do not attempt.
  • The predictor's reliance on coding, safety, instruction following, and parameter count suggests a cheap pre-screen for new base models: compute those five reported numbers before committing GPU-hours to reward-model training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper investigates how the choice of base model affects downstream reward modeling performance. The authors train 40 chat models (0.49B to 10.3B parameters) from multiple families using two objectives — Bradley-Terry with binary preferences on HelpSteer2-Preference and multi-attribute regression on HelpSteer2 — and evaluate all resulting reward models on RewardBench. They report relative gains of up to 14% over the commonly used Llama-3.x baselines in matched parameter-size groups, analyze correlations between 33 standard benchmarks and RewardBench scores, and propose an Elastic Net regression that combines a small set of benchmark features for model selection, claiming an average +18% improvement in top-5/10 coverage over single benchmarks. The paper also studies post-training stages using publicly available Llama-3.1-Tulu-3 checkpoints, finding that SFT helps (+15.5%) while subsequent DPO/RLVR steps hurt (3-5%), and estimates pre-training data distributions of several models using length-normalized sequence probabilities, reporting a +1.5% improvement in reward-model prediction MAE when these distribution features are added.

Significance. If the results hold, the paper makes a useful practical contribution by identifying base model choice as a high-leverage hyperparameter for reward modeling. The study is broad in scope: 40 models, two training objectives, four RewardBench subcategories, and multiple training-stage comparisons. The authors also provide a helpful independent verification of RewardBench labels (Appendix A) and make concrete experimental details available, including full per-model results in the appendices. The main risk is that the model-selection claim is evaluated in-sample, so the headline +18% coverage improvement is not yet shown to transfer to new base models; additionally, the absence of repeated-seed variance reporting makes the size of the base-model effect less certain. These issues are fixable within the scope of the manuscript, so the work is promising but needs revision.

major comments (3)
  1. [Section 4.2 and Figure 6] The central model-selection claim, an average +18% improvement in top-5/10 coverage, is evaluated in-sample. Section 4.2 states that a 10-fold cross-validation is run only to search Elastic Net hyperparameters, after which "we fit a model over all samples using the best hyperparameters." The coverage plotted in Figure 6 is therefore computed for the same 40 models used to fit the final regression, so the comparison measures how well the fitted model ranks its own training data, not how well benchmark features predict RewardBench for a new base model. Because this claim appears in the abstract and in contribution 3, the authors should provide a held-out model-level evaluation, for example by leaving out whole models or model families in an outer cross-validation loop and reporting coverage on the held-out folds, or by using an explicit train/validation split. The feature-selection process should also be nested inside the validation procedure, since the five selected features were identified after inspecting the full dataset.
  2. [Section 3.1 and Section 3.3] The headline result of 3-14% performance gains from base model choice rests on a single training run per base model. Section 3.1 describes one learning-rate grid search per model and checkpoint selection on the HelpSteer2 validation set, with no repeated seeds or confidence intervals reported in Section 3.3 or in Tables 4-5. If run-to-run variance from random seeds, learning-rate sensitivity, or checkpoint selection is comparable to the observed gaps, the gains cannot be cleanly attributed to base model choice. I recommend repeating training for at least the headline comparison pairs and a random subset of other models with multiple seeds, and reporting the mean and standard deviation (or confidence intervals) so the reader can assess whether the observed gaps exceed optimization noise. The consistent family-level patterns (Gemma-2 and Qwen2.5 outperforming Llama-3.x across both objectives) partially mitigate this concern, but the size of the claimed gains needs variance quantification.
  3. [Section 5.2, last paragraph] The reported +1.5% MAE improvement from adding estimated pre-training distribution features (from 3.2% to 1.7%) appears to use the same in-sample regression setup as Section 4.2: the regression is fit on all models and the MAE is computed on the same models. The paragraph also does not specify how many models are included in this analysis, and the pre-training distribution estimates are only available for a subset of the 40 models. As written, this improvement may reflect overfitting rather than predictive benefit. The authors should state whether the reported MAE is in-sample or cross-validated, and if the latter, describe the exact folds and model subset; if the former, they should add a held-out evaluation before claiming that pre-training distributions reduce prediction error.
minor comments (4)
  1. [Figure captions] The coverage filtering thresholds are inconsistent across figures: Figure 4 retains benchmarks with at least 0.4 and 0.7 coverage at k=5 and k=10, while Figures 9 and 10 in Appendix B use 0.4 and 0.6. Please reconcile the thresholds and state the final criterion clearly.
  2. [Section 4.1, Significance Test] The significance test in Eq. (8) is applied to many pairwise correlations, but no multiple-testing correction is mentioned. The statement that "most of the benchmarks' correlations have statistical significance" should be qualified in light of the number of tests performed, or the analysis should include a correction such as Benjamini-Hochberg.
  3. [Throughout] There are several typos and minor wording issues, including "RewarcBench" in Section 4.2, "LLama" in the introduction, and the phrase "finding their similar performances" in Appendix E. A careful proofread would improve presentation.
  4. [Section 3.3] The text says results show "potentially high variances in performance within groups of models with similar sizes," but no variance estimates are provided anywhere in the paper. Please either report such estimates or rephrase to refer to observed spread across models rather than statistical variance.

Circularity Check

2 steps flagged · score 6.0 of 10

Partial circularity: the +18% model-selection gain and +1.5% MAE improvement are in-sample fits, not held-out predictions.

  1. fitted input called prediction [Section 4.2, Regression Analysis, Setup and Results (Figures 5-6)]
    "we run a 10-fold cross-validation experiment on an Elastic Net model, searching over the following hyperparameters: ... Then, we fit a model over all samples using the best hyperparameters. ... As evident, the trained model significantly improves the coverage in lower Ks, mitigating the critical problem of using standalone benchmarks. These results prove our hypothesis, showcasing the predictability of reward modeling performance from a low-dimensional vector of prior results."

    The Elastic Net's target is the RewardBench score of the same 40 models used to fit it. The 10-fold CV is used only for hyperparameter selection, and the final model is fit on all samples; the coverage in Figure 6 is then computed between the fitted values on those same 40 training models and their actual RewardBench ranks. Thus the reported +18% top-5/10 improvement over single benchmarks is an in-sample measure of fit quality, not a prediction for new base models. A regression trained on the evaluation target will naturally rank its training points well, so the claimed model-selection benefit is not established as a predictive result.

  2. fitted input called prediction [Section 5.2, Pre-training Results]
    "Moreover, we expand our regression analysis (see Section 4.2) by adding the average scores of the categories to the already established five features (see Figure 5). Our experiments show that compared to adding these features improves the mean absolute error by +1.5% (from 3.2% to 1.7%), compared to only using the original five features, which showcases the untapped potential of the pre-training data distributions."

    The +1.5% MAE improvement compares in-sample errors of two Elastic Net fits on the same 40 models: one with the original five features and one additionally using estimated pre-training distribution scores. Because both models are fit on all samples and evaluated on those same samples, the reported MAE is training error rather than prediction error. Adding features can improve in-sample fit without improving generalization to new base models, so the claim that pre-training distributions 'reduce performance prediction error' is not supported by an out-of-sample evaluation.

full rationale

The central base-model comparison (Section 3.3) is self-contained and not circular: the authors train reward models from public checkpoints with a fixed protocol and evaluate on RewardBench, yielding externally falsifiable measurements. The correlation analysis (Section 4.1) is descriptive statistics on the measured outcomes, not a prediction. The post-training analysis (Section 5.1) and RewardBench verification (Appendix A) are also independent of the paper's fitted regression. The circularity is confined to Section 4.2 and its extension in Section 5.2: the Elastic Net is fit on all 40 models and then evaluated on the same 40 models, so the headline +18% selection gain and the +1.5% MAE improvement are in-sample descriptions of the fit, not predictive validations. Since the base-model gains remain independently supported, the paper is only partially circular, not wholly reducible to its inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. The central claims rest on empirical measurements and regression fits, with hand-chosen coverage thresholds and the assumption that single training runs isolate base model effects.

free parameters (3)
  • Elastic Net regression coefficients and hyperparameters = not fully reported; 5 benchmarks selected
    The model selection predictor in Section 4.2 is fitted to the 40 reward scores; reported +18% coverage is computed on the same data.
  • Coverage filter thresholds = 0.4 at k=5 and 0.7 at k=10
    Hand-chosen cutoffs in Section 4.1 that determine which benchmarks are analyzed; affects the correlation and coverage narrative.
  • Per-model learning rates and merge vectors = searched per model, not reported
    Training hyperparameters selected on validation sets for each base model; if these searches are noisy, the 3-14% gaps are not cleanly attributable to base model.
assumptions (4)
  • domain assumption RewardBench accuracy is a valid proxy for reward model quality
    All conclusions about base model quality rest on RewardBench as ground truth; Appendix A verifies 98% agreement on 50 disputed samples, but this is a limited check.
  • domain assumption A single training run per base model with validation-based checkpoint selection yields stable estimates of base model quality
    Section 3.1 trains each model once (per objective) with a learning rate search; no multiple seeds or confidence intervals are reported, so observed gaps may include training noise.
  • domain assumption Crystal trained exactly once on SlimPajama can serve as ground truth for membership inference
    Section 5.2 uses Crystal's sequence probabilities as ground truth presence scores, assuming its training data exposure is known and static.
  • domain assumption Estimated pre-training composition for the largest family member transfers to smaller models trained on the same token budget
    Stated in Section 5.2: the paper reuses extracted distribution from the largest member if the model was trained on the same amount of data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Systematic Analysis of Base Model Choice for Reward Modeling." pith.science (2026). https://pith.science/paper/WBGO6T5N

@misc{pith2026250510775,
  author       = {Pith},
  title        = {Pith review of: A Systematic Analysis of Base Model Choice for Reward Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WBGO6T5N}},
  note         = {Machine review of arXiv:2505.10775}
}
abstract

Reinforcement learning from human feedback (RLHF) and, at its core, reward modeling have become a crucial part of training powerful large language models (LLMs). One commonly overlooked factor in training high-quality reward models (RMs) is the effect of the base model, which is becoming more challenging to choose given the rapidly growing pool of LLMs. In this work, we present a systematic analysis of the effect of base model selection on reward modeling performance. Our results show that the performance can be improved by up to 14% compared to the most common (i.e., default) choice. Moreover, we showcase the strong statistical relation between some existing benchmarks and downstream performances. We also demonstrate that the results from a small set of benchmarks could be combined to boost the model selection ($+$18% on average in the top 5-10). Lastly, we illustrate the impact of different post-training steps on the final performance and explore using estimated data distributions to reduce performance prediction error.

Figures

Figures reproduced from arXiv: 2505.10775 by the authors.

Figure 1
Figure 1. Ratio of the base models used in the top 30 entries of RewardBench (Dec 2024). Almost all the entries are trained on top of a small set of base models (e.g., Llama-3.x models comprise 50% of the entries). more than 50% of the top 30 entries (see [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Reward Modeling Performance Gains. Relative gains are illustrated concerning the Llama-3.x model (marked as red) within the same group. Bradley-Terry (BT) (Bradley and Terry, 1952; Ziegler et al., 2019) model. The underlying as￾sumption of BT is that for a pair of responses Y = (y1, y2), the human preference distribution ρ ∗ is generated from a latent reward function r ∗ (x, y), which we only have indirect access to… view at source ↗
Figure 3
Figure 3. Statistical Correlation w.r.t. Reward Modeling Performance. The subset benchmarks of Open LLM Leaderboard v2 (v1) are denoted with an ‡ (†). Text Colors: Red → Aggregate benchmark, Green → Training metric. ANLI (Williams et al., 2022) consistently beating other benchmarks across different subcategories. Significance Test. We test the significance of the correlation coefficient with the following statistic: tc = r √ … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Benchmark’s Coverage. We only retain benchmarks with at least 0.4 and 0.7 coverage at k = 5 and k = 10, respectively. real-world search where we need high coverage at higher ranks, we filter out any benchmark with less than 0.4 and 0.7 coverage at k = 5 and k = 10, res…
Figure 6
Figure 6. Figure 6: Benchmarks vs. Predicted Score Coverage. We only retain benchmarks with at least 0.4 and 0.7 coverage at k = 5 and k = 10, respectively. critical problem of using standalone benchmarks. These results prove our hypothesis, showcasing the predictability of reward modelin…
Figure 7
Figure 7. Figure 7: Estimated Pre-training Data Distributions. Crystal (Liu et al., 2024d) represents our ground truth, as it has seen the entire SlimPajama dataset in the pre-training phase exactly once [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Jensen-Shannon Distance. The values are based on the scores from the entire dataset. Results [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Benchmark’s Coverage. We only retain benchmarks with at least 0.4 and 0.6 coverage at k = 5 and k = 10, respectively. A RewardBench as Ground Truth Given the heavy reliance of our work on Reward￾Bench, we conduct an independent verification of the preferences. Specific…
Figure 11
Figure 11. Figure 11: Bradley-Terry vs. Regression Perfor￾mance Difference. A positive value indicates a better performance on the Regression method. previous works have briefly compared Bradley￾Terry vs. Regression training (Wang et al., 2024c), finding their similar performances on ∼70B …
Figure 12
Figure 12. Figure 12: Principal Component’s Weights [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: PCA Explained Variance. We find that the top 5 PCs explain ∼96.8% of the variance; hence, the benchmark-model matrix is low-dimensional. mance match on 70B scale models, consistent with previous findings (see Appendix D for more de￾tails). This observation suggests th…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

85 extracted references · 7 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024 a . Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S \'e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024 b . Phi-4 technical report. arXiv preprint arXiv:2412.08905

    arXiv 2024

  5. [5]

    Arash Ahmadian, Chris Cremer, Matthias Gall \'e , Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet \"U st \"u n, and Sara Hooker. 2024. https://doi.org/10.18653/v1/2024.acl-long.662 Back to basics: Revisiting REINFORCE -style optimization for learning from human feedback in LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Co...

    doi:10.18653/v1/2024.acl-long.662 2024

  6. [6]

    Anthropic. 2024. Meet claude. https://www.anthropic.com/claude. Accessed: 2024-11-25

    2024

  7. [7]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  8. [8]

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073

    arXiv 2022

Show all 85 references
  1. [9]

    Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. 2024. https://doi.org/10.18653/v1/2024.acl-long.419 MARS : Meaning-aware response scoring for uncertainty estimation in generative LLM s . In Proceedings of th...

    2024 doi

  2. [10]

    Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023. Open llm leaderboard (2023-2024). https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard

    2023

  3. [11]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432--7439

    2020

  4. [12]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

    1952

  5. [13]

    David Cecchini, Arshaan Nazir, Kalyan Chakravarthy, and Veysel Kocaman. 2024. https://doi.org/10.18653/v1/2024.trustnlp-1.11 Holistic evaluation of large language models: Assessing robustness, accuracy, and toxicity for real-world applications . In Proceedings of the 4th Works...

    2024 doi

  6. [14]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  7. [15]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1300 B ool Q : Exploring the surprising difficulty of natural yes/no questions . In Proceedings of the 2019 Conference of the North A ...

    2019 doi

  8. [16]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  9. [17]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  10. [18]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. Ultrafeedback: boosting language models with scaled ai feedback. In Proceedings of the 41st International Conference on Mac...

    2024

  11. [19]

    Smith, and Matt Gardner

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. https://doi.org/10.18653/v1/2021.naacl-main.365 A dataset of information-seeking questions and answers anchored in research papers . In Proceedings of the 2021 Conference of the North Amer...

    2021 doi

  12. [20]

    Nicolai Dorka. 2024. Quantile regression for distributional reward models in rlhf. arXiv preprint arXiv:2409.10164

    2024 arXiv

  13. [21]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. https://doi.org/10.18653/v1/N19-1246 DROP : A reading comprehension benchmark requiring discrete reasoning over paragraphs . In Proceedings of the 2019 Conference of the North A ...

    2019 doi

  14. [22]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  15. [23]

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

    2024

  16. [24]

    Leo Gao, John Schulman, and Jacob Hilton. 2023 a . https://proceedings.mlr.press/v202/gao23h.html Scaling laws for reward model overoptimization . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, p...

    2023

  17. [25]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  18. [26]

    Pengzhi Gao, Liwen Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-industry.25 Learning multilingual sentence representations with cross-lingual consistency regularization . In Proceedings of the 2023 Conference on Empirical Method...

    2023 doi

  19. [27]

    Gemini Team . 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  20. [28]

    Srishti Gureja, Lester James V Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, and Marzieh Fadaee. 2024. M-rewardbench: Evaluating reward models in multilingual settings. arXiv preprint arXiv:2410.15522

    2024 arXiv

  21. [29]

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. https://doi.org/10.18653/v1/2022.acl-long.234 T oxi G en: A large-scale machine-generated dataset for adversarial and implicit hate speech detection . In Proceedings of the 60th A...

    2022 doi

  22. [30]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In International Conference on Learning Representations

    2021

  23. [31]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Sim...

    2022

  24. [32]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654

    2024 arXiv

  25. [33]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    2021

  26. [34]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  27. [35]

    Gyuwan Kim, Yang Li, Evangelia Spiliopoulou, Jie Ma, Miguel Ballesteros, and William Yang Wang. 2024. Detecting training data of large language models via expectation maximization. arXiv preprint arXiv:2410.07582

    2024

  28. [36]

    Tomasz Korbak, Ethan Perez, and Christopher Buckley. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.77 RL with KL penalties is better viewed as B ayesian inference . In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1083--1091, Abu Dhabi, U...

    2022 doi

  29. [37]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  30. [38]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

    2024 arXiv

  31. [39]

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024 b . Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787

    2024 arXiv

  32. [40]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2022 doi

  33. [41]

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024 a . Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451

    2024 arXiv

  34. [42]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. 2023. https://openreview.net/forum?id=1qvx610Cu7 Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation . In Thirty-seventh Conference on Neural Informa...

    2023

  35. [43]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 b . https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics,...

    2024 doi

  36. [44]

    Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2024 c . Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410.16184

    2024 arXiv

  37. [45]

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Roberto Iriondo,...

    2024

  38. [46]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In International Conference on Learning Representations

    2019

  39. [47]

    Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. 2024. Uncertainty-aware reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.00847

    2024 arXiv

  40. [48]

    Andrey Malinin and Mark Gales. 2021. https://openreview.net/forum?id=jN5y-zb5Q7m Uncertainty estimation in autoregressive structured prediction . In International Conference on Learning Representations

    2021

  41. [49]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. https://doi.org/10.18653/v1/D18-1260 Can a suit of armor conduct electricity? a new dataset for open book question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P...

    2018 doi

  42. [50]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.154 C row S -pairs: A challenge dataset for measuring social biases in masked language models . In Proceedings of the 2020 Conference on Empirical Methods in Na...

    2020 doi

  43. [51]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2024. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656

    2024 arXiv

  44. [52]

    OpenAI. 2024. Introducing openai o1. https://openai.com/o1. Accessed: 2024-11-25

    2024

  45. [53]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  46. [54]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

    2019

  47. [55]

    Felipe Maia Polo, Seamus Somerstep, Leshem Choshen, Yuekai Sun, and Mikhail Yurochkin. 2024. Sloth: scaling laws for llm skills to predict multi-benchmark performance across families. arXiv preprint arXiv:2412.06540

    2024

  48. [56]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. https://openreview.net/forum?id=HPuSIXJaa9 Direct preference optimization: Your language model is secretly a reward model . In Thirty-seventh Conference on Neural Infor...

    2023

  49. [57]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don`t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 78...

    2018 doi

  50. [58]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. https://openreview.net/forum?id=Ti67584b98 GPQA : A graduate-level google-proof q&a benchmark . In First Conference on Language Modeling

    2024

  51. [59]

    Maddison, and Tatsunori Hashimoto

    Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. 2024. https://openreview.net/forum?id=On5WIN7xyD Observational scaling laws and the predictability of langauge model performance . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  52. [60]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  53. [61]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. https://doi.org/10.18653/v1/D19-1454 Social IQ a: Commonsense reasoning about social interactions . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9...

    2019 doi

  54. [62]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  55. [63]

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. https://openreview.net/forum?id=zWqr3MQuNs Detecting pretraining data from large language models . In The Twelfth International Conference on Learning Re...

    2024

  56. [64]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. Slimpajama: A 627b token cleaned and deduplicated version of redpajama

    2023

  57. [65]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf Learning to summarize with human fee...

    2020

  58. [66]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. https://doi.org/10.18653/v1/2023.findings-acl.824 Challenging BIG -bench tasks and whether chain-of-thought c...

    2023 doi

  59. [67]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  60. [68]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  61. [69]

    Ryan Teknium, Jeffrey Quesnelle, and Chen Guang. 2024. Hermes 3 technical report. arXiv preprint arXiv:2408.11857

    2024 arXiv

  62. [70]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024 a . https://doi.org/10.18653/v1/2024.findings-emnlp.620 Interpretable preferences via multi-objective reward modeling and mixture-of-experts . In Findings of the Association for Computational Linguistics: E...

    2024 doi

  63. [71]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024 b . https://openreview.net/forum?id=y10DM6R2r3 MMLU -pro: A m...

    2024

  64. [72]

    Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. 2024 c . Helpsteer2-preference: Complementing ratings with preferences. arXiv preprint arXiv:2410.01257

    2024 arXiv

  65. [73]

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024 d . Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673

    2024 arXiv

  66. [74]

    Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. 2024 e . https://doi.org/10.18653/v1/2024.naacl-long.185 H elp S teer: Multi-attribute helpfulness datase...

    2024 doi

  67. [75]

    Adina Williams, Tristan Thrush, and Douwe Kiela. 2022. https://aclanthology.org/2022.scil-1.3/ ANLI zing the adversarial natural language inference dataset . In Proceedings of the Society for Computation in Linguistics 2022, pages 23--54, online. Association for Computational ...

    2022

  68. [76]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  69. [77]

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464

    2024 arXiv

  70. [78]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  71. [79]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/v1/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4...

    2019 doi

  72. [80]

    Anqi Zhang and Chaofeng Wu. 2024. Adaptive pre-training data detection for large language models via surprising tokens. arXiv preprint arXiv:2407.21248

    2024 arXiv

  73. [81]

    Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.300 Pretraining data detection for large language models: A divergence-based calibration method . In Proceedings of the 2024 Conference...

    2024 doi

  74. [82]

    Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, and Quanquan Gu. 2024 b . General preference modeling with preference representations for aligning language models. arXiv preprint arXiv:2410.02197

    2024 arXiv

  75. [83]

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.149 AGIE val: A human-centric benchmark for evaluating foundation models . In Findings of the Association for...

    2024 doi

  76. [84]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

  77. [85]

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

    2019 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.