Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A language model aligns better when its two internal reward critics are forced to agree.

desk verdict A plausible, well-engineered method for self-rewarding alignment with large AlpacaEval gains, but the mechanism evidence is filtered and single-run; worth refereeing, not worth believing at face value. read the letter →

arxiv 2502.08922 v1 pith:556YH6CG submitted 2025-02-13 cs.AI

Xin Zhou , Yiwen Guo , Ruotian Ma , Tao Gui , Qi Zhang , Xuanjing Huang This is my paper · ORCID
classification cs.AI
keywords self-rewardinglanguagemodelsLLM-as-a-judgedirectpreferenceoptimizationinternalrewardconsistencyiterativeDPOdataqualityalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Self-rewarding language models use the model itself as a judge to produce training preferences, but the paper reports that the two reward signals inside one model—the LLM-as-a-Judge and the implicit reward hidden in the DPO objective—disagree on about half of preference pairs. The paper proposes SCIR, which trains those two internal reward views to agree and then feeds only the pairs on which they already agree into iterative DPO. The central claim is that this self-consistency both improves alignment and makes the internal rewards more accurate. On Mistral-7B the length-controlled AlpacaEval 2.0 win rate rises from 10.81 to 24.92 across three iterations, above every baseline tested including an external reward model.

What carries the argument

The load-bearing object is a consistency regularizer between two preference distributions computed from the same model. For an unlabeled pair $(x, y_1, y_2)$, the implicit DPO view is $P_{\mathrm{irm}} = \sigma\!\big(\beta \log \frac{\pi_\theta(y_1\mid x)}{\pi_{\mathrm{ref}}(y_1\mid x)} - \beta \log \frac{\pi_\theta(y_2\mid x)}{\pi_{\mathrm{ref}}(y_2\mid x)}\big)$ and the generative view $P_{\mathrm{grm}}$ is the judge-token probability under averaged pairwise judge prompts. The overall objective is $I(r_1 = r_2 = \cdots = r_n)\,L_{\mathrm{DPO}} + \alpha L_{\mathrm{consistency}}$, where $L_{\mathrm{consistency}}$ is a symmetric KL term with stop-gradient targets, entropy regularization $H(P)$, and confidence thresholds $\tau$ that activate the loss only when a view's top probability exceeds $\tau$. This mechanism pulls the two reward views toward each other and away from the trivial 0.5 equilibrium, while the indicator in front of the DPO term turns data selection into a dynamic filter that keeps only currently consistent pairs for preference optimization.

What would settle it

A concrete test: train the SCIR pipeline on Mistral-7B-Instruct but replace the consistency filter with a filter that keeps pairs ranked confidently by the implicit reward model alone, holding all other hyperparameters fixed; if the length-controlled AlpacaEval win rate does not drop, mutual agreement between the two views is not the mechanism producing the gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that a preference label is trustworthy only if it is invariant across the model's internal reward views, and that a well-aligned model should be self-consistent across those views. SCIR operationalizes this with two views: the generative reward model, which is the LLM-as-a-Judge probability that response $y_1$ beats $y_2$, and the implicit reward model $P_{\mathrm{irm}} = \sigma\!\big(\beta \log \frac{\pi_\theta(y_1\mid x)}{\pi_{\mathrm{ref}}(y_1\mid x)} - \beta \log \frac{\pi_\theta(y_2\mid x)}{\pi_{\mathrm{ref}}(y_2\mid x)}\big)$ derived from the DPO objective. Each training step minimizes a symmetric KL consistency loss with entropy regularization and confidence masking so the two views become both consistent and confident, and the DPO term is applied only to pairs where all internal views agree. Across three iterations on Mistral-7B this raises the length-controlled AlpacaEval 2.0 win rate from 10.81 to 24.92 and the MT-Bench score from 5.39 to 6.18, while the consistency rate between the two reward views climbs and RewardBench accuracy of the agreeing subset is higher than either view alone.

Load-bearing premise

Agreement between the generative judge and the implicit DPO reward is a reliable sign that a preference label is correct, so that enforcing agreement and training only on agreeing pairs improves data quality rather than merely shrinking the training set.

Editorial extensions

If this is right

  • A 7B model can match or exceed an external 8B reward model at alignment without any human preference labels in the loop.
  • Length-controlled win rate, which removes the confound of verbosity, continues to improve across iterations, suggesting the gain is not merely longer outputs.
  • The consistency rate between the two internal reward views rises with iterations and the RewardBench accuracy of their agreeing predictions is higher than either view alone, so the method improves the reward signal itself, not just the downstream alignment.
  • The same pipeline transfers from a weak SFT model to Mistral-7B-Instruct, raising the length-controlled AlpacaEval win rate by roughly 12 points, so the benefit is not specific to one initialization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the paper's own RewardBench analysis shows the generative judge is near chance on Mistral-7B, part of the gain may come from using agreement as a confidence filter that mostly keeps pairs the stronger implicit reward got right; a testable version would compare SCIR against filtering on implicit-reward confidence alone.
  • Editorial inference: enforcing consistency with a symmetric KL objective can in principle drag the stronger view toward the weaker one, so monitoring RewardBench accuracy of each view separately across iterations would show whether consistency training genuinely improves the judge or just concentrates training on easy pairs.
  • Editorial inference: the same consistency-with-selection recipe should apply to any set of internal reward views, such as multiple judge prompts, multiple reference models, or hidden-layer probes, making SCIR a general instance of ensembling a model's own preferences rather than a special trick for two views.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies self-rewarding language models (SRLMs), in which an LLM generates its own preference data via LLM-as-a-Judge. It observes that the generative reward model (GRM) and the implicit DPO reward model (IRM) of the same model often disagree (~50% in Table 1). It proposes Self-Consistent Internal Rewards (SCIR): during each training step, both reward models produce preference probabilities, are trained toward each other with a confidence-masked, entropy-regularized symmetric KL loss (Eq. 6), and DPO is applied only to pairs on which they agree (Eq. 7). Additional components include multiple judge prompts, length regularization, and an adaptive reference model. Experiments on Mistral-7B and Mistral-7B-Instruct show large AlpacaEval 2.0 LC gains for SCIR relative to SRLM and external reward model baselines, and RewardBench accuracy plots are used to argue that consistency improves reward modeling.

Significance. If the empirical claims hold, SCIR is a useful contribution: it is a simple, training-efficient way to use internal consistency to filter self-generated preference data, with ablations showing that each component matters (Table 4). The paper is transparent about baselines and implementation details, and the use of length-controlled metrics mitigates a known confound. However, the strength of the evidence is limited by the absence of any variance or significance reporting and by the RewardBench evaluation being restricted to filtered subsets; the mechanism linking consistency to correctness is not cleanly established. These issues are fixable and the central empirical direction is worth pursuing.

major comments (3)
  1. [Sec. 4.2, Tables 2-3] All reported results are single runs: no seeds, confidence intervals, or significance tests are given. The headline claim that SCIR outperforms baselines by a notable margin rests on point estimates such as LC 24.92 versus 15.91 on Mistral-7B and 35.02 versus 24.01 on Mistral-7B-Instruct. Given the stochasticity of iterative DPO and of GPT-judge-based AlpacaEval evaluation, these gaps could be within run-to-run noise. Please provide multiple seeds or bootstrap confidence intervals for the main comparisons and for the Table 4 ablations.
  2. [Sec. 4.4, Fig. 3] RewardBench accuracy is reported only on filtered subsets: invalid pairs are excluded and the Consistency curves are computed only on pairs where IRM and GRM agree. This is the same event used for data selection in Eq. 7, so higher accuracy on the Consistency subset can arise from selection alone and is not an apples-to-apples comparison with SRLM-IRM/SRLM-GRM on the full valid set. On Mistral-7B the GRM is near chance (about 50% in Fig. 3), so a noise judge would agree with the IRM about half the time and the agreeing subset could appear accurate without any genuine reward signal. Please report full-benchmark accuracy counting ties and invalid predictions as incorrect, report the coverage of each subset, and include a random-judge agreement oracle as a control.
  3. [Sec. 3.1-3.2, Sec. 4.3] The consistency rate in Fig. 2 is by construction the objective being optimized by Eq. 6 and the selection rule in Eq. 7, so it is not independent evidence that consistency marks correct preferences. The mechanistic premise of the paper—that agreement between GRM and IRM is a reliable preference signal—is fragile when one view is near chance: the symmetric KL with stop-gradient can pull the stronger IRM toward the weaker judge's confident errors or sharpen shared biases such as length and position preference. Please provide a direct test of the premise, e.g., RewardBench accuracy of Ours-Consistency on the full valid set versus a control that applies the same consistency filter with a deliberately uninformative judge, and report agreement rates separately for correct and incorrect IRM predictions.
minor comments (5)
  1. [Sec. 4.1 / Table 2 / App. C.2] The external reward model is called Skywork-Reward-8B in the text, Skywork-reward-7B in Table 2, and Skywork-reward-7B in Appendix C.2; the exact checkpoint used should be stated consistently because the baseline comparison depends on it.
  2. [Abstract / Sec. 4.2] The intro claims 'a 14% improvement in length-controlled win rate' while the results section says the LC win rate rises 'from 10.81% to 24.96%' (Table 2 reports 24.92); please make the numbers consistent and specify whether the change is in percentage points or relative.
  3. [Sec. 4.3] The consistency rate in Fig. 2 is computed only on the subset of pairs for which the GRM predictions are already consistent across the four judge prompts; this conditional definition should be stated in the caption or in the main text.
  4. [Sec. 3.1, Eq. 6] The notation I(P > tau) is ambiguous because P is a probability of one outcome; please clarify that the indicator applies to the highest probability (or to both directions) and justify the threshold choice tau = 0.7.
  5. [App. A] The adaptive reference model chooses the reference that maximizes the odds ratio, which makes the DPO constraint stricter but does not by itself guarantee consistency of the two IRM variants; please either provide a formal argument or soften the claim, and ideally report the effect of this component on IRM consistency separately from the Table 4 ablation.

Circularity Check

1 steps flagged · score 4.0 of 10

Alignment gains are externally grounded, but the reward-modeling evidence for "consistency = reliability" is evaluated on the agreement-filtered subset, making that supporting claim partially self-referential.

  1. other [Section 4.4 (Reward Modeling Ability), Figure 3; Equation 7 (DCPO)]
    "We directly use the average accuracy between the predicted results and the standard labels as the metric... We show the accuracy of the GRM and IRM using different methods and additionally display the accuracy when the predictions of GRM and IRM are consistent (Consistency)."

    Equation 7 trains DPO only on pairs satisfying I(r1 = r2 = ... = rn), i.e., on the event that IRM and GRM agree. Figure 3's 'Consistency' accuracy is measured on exactly the same agreement event: pairs where IRM and GRM predict the same label. The conclusion that 'consistency leads to more reliable preferences' is therefore not an independent test; it conditions on the filter used to define the data. Because the generative reward model is near chance on Mistral-7B (about 50% in Figure 3), agreement can arise by chance, and the accuracy of the agreeing subset can be high for any model due to selection alone. The paper does not report full-benchmark accuracy or selection coverage, so the reward-modeling improvement is not established as a causal effect of consistency.

full rationale

No self-citation chain, uniqueness theorem, or imported ansatz is load-bearing. The central alignment result is measured on external benchmarks (AlpacaEval 2.0 LC win rate rises from 10.81 to 24.92; MT-Bench score rises from 5.39 to 6.18), and the ablation in Table 4 shows that removing the consistency components degrades performance, so the main claim has independent empirical support. The circular component is confined to the reward-modeling analysis: the paper equates 'consistent predictions' with 'more reliable preferences' using an accuracy computed on the agreement-filtered subset, which is the same selection rule as Equation 7. This conditioning does not invalidate the external alignment results but weakens the secondary claim about reward modeling. The consistency-rate improvement in Figure 2 is also partly in-sample since it reflects the quantity the consistency loss promotes, but I do not treat that as a separate circular step; the score of 4 reflects one supporting 'prediction' being evaluated on its own selection criterion rather than a fully circular derivation.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard DPO theory, the validity of LLM-as-a-Judge as a preference source, and the unproven premise that agreement between the two internal reward models indicates label correctness. The method introduces no new entities but does add two hyperparameters (tau, alpha) and a length penalty, plus a per-model schedule for when the penalty is enabled.

free parameters (5)
  • alpha (consistency loss weight) = 1.0
    Controls strength of L_consistency in Eq. 7; set from preliminary experiments, no sensitivity analysis.
  • tau (confidence threshold) = 0.7
    Gates consistency loss in Eq. 6; set from preliminary experiments.
  • alpha_l (length regularization coefficient) = 0.02
    Subtracts alpha_l * |y| from reward logits; enabled in different iterations for the two base models based on preliminary trials.
  • beta (DPO temperature) = 0.1
    Standard DPO hyperparameter from prior work; not tuned here.
  • length-regularization start iteration = iteration 2 (Instruct), iteration 3 (v0.3)
    Chosen per model from preliminary results, a post-hoc schedule.
assumptions (6)
  • standard math DPO implicit reward r(x,y) = beta log(pi_theta/pi_ref) + beta log Z(x) is a valid reward for preference optimization
    Equation 1-3, taken from Rafailov et al. 2024.
  • standard math Bradley-Terry model of preferences holds for pairwise judgments
    Equation 2, standard in DPO.
  • domain assumption LLM-as-a-Judge provides a meaningful preference signal
    The method relies on judge-prompt outputs as one of two reward models; the paper's own data show this judge can be near chance on Mistral-7B.
  • domain assumption Agreement between the generative and implicit reward models indicates a correct preference label
    Core motivation in Section 3 and Section 4.4; not proven, and fragile if one reward model is random.
  • domain assumption Randomly sampling two responses from the current policy produces useful preference pairs for DPO
    SCIR forms 4,000 unlabeled pairs by random pairing of four candidate responses (Section 4.1).
  • ad hoc to paper The consistency loss with stop-gradient and confidence masking avoids collapse to the trivial 0.5 solution
    Claimed in Section 3.1 without formal proof; relies on entropy regularization and thresholding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models." pith.science (2026). https://pith.science/paper/556YH6CG

@misc{pith2026250208922,
  author       = {Pith},
  title        = {Pith review of: Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/556YH6CG}},
  note         = {Machine review of arXiv:2502.08922}
}
read the original abstract

Aligning Large Language Models (LLMs) with human preferences is crucial for their deployment in real-world applications. Recent advancements in Self-Rewarding Language Models suggest that an LLM can use its internal reward models (such as LLM-as-a-Judge) \cite{yuanself} to generate preference data, improving alignment performance without costly human annotation. However, we find that different internal reward models within the same LLM often generate inconsistent preferences. This inconsistency raises concerns about the reliability of self-generated preference data, hinders overall alignment performance, and highlights the need for further research to ensure reliable and coherent alignment with human preferences. To address this limitation, we propose Self-Consistent Internal Rewards (SCIR), a novel framework designed to enhance consistency among internal reward models during training. In each training step, we collect preference predictions from multiple pre-defined internal reward models and enforce consistency and confidence through an inconsistency penalty mechanism, thereby improving the reliability of these internal reward models. We selectively use data with consistent predictions for preference optimization, ensuring the quality of the preference data. By employing self-consistent internal rewards, our method significantly improves the alignment performance and reward modeling capability of LLMs, outperforming baseline methods by a notable margin.

Figures

Figures reproduced from arXiv: 2502.08922 by the authors.

Figure 1
Figure 1. An overview of our framework. For each iteration, the LLM Mt generates responses for the prompts in the prompt pool, constructing unlabeled preference pairs. Then these pairs are used to optimize Mt via Self-Consistent Intern Rewards (SCIR) Training. In each training step, the model’s implicit DPO reward model and generative reward model predict the preference probabilities for each unlabeled preference pair. We use… view at source ↗
Figure 2
Figure 2. Consistency rate of internal reward models. Mt is the model after the t iterations. New Data and Trained Data refer to the preference data from the t-th and the (t-1)-th iteration, respectively. closely aligned with tasks in MMLU and GSM8K, resulting in minimal impact on the model’s general ability. 4.3. Consistency of Internal Reward Models We follow the experimental setup in Section 2.3, using the model’s internal… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning

    cs.AI 2025-06 conditional novelty 6.0 of 10

    CoVo trains LLMs with a self-generated reward based on the consistency and volatility of intermediate reasoning states, matching supervised RL performance without external labels.

  2. Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future

    cs.CL 2025-08 conditional novelty 5.0 of 10

    Anchoring rejected responses to the initial model and choosing responses from a future model raises AlpacaEval 2.0 win rate from 19.69 to 29.44 for Llama3.1-8B.

Reference graph

Works this paper leans on

49 extracted references · 15 canonical work pages · cited by 2 Pith papers

  1. [1]

    Meta-rewarding language models: Self-improving alignment with LLM -as-a-meta-judge

    Anonymous. Meta-rewarding language models: Self-improving alignment with LLM -as-a-meta-judge. In Submitted to The Thirteenth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=lbj0i29Z92. under review

    work page 2024

  2. [2]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022 a

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, ...

    2022

  3. [3]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022 b

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, ...

    arXiv 2022

  4. [4]

    E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., ...

    work page 2022

  5. [5]

    A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M

    Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  6. [6]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  7. [7]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    2020

  8. [8]

    H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al

    Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning, a

    work page

Show all 49 references
  1. [9]

    Discovering latent knowledge in language models without supervision

    Burns, C., Ye, H., Klein, D., and Steinhardt, J. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, b

  2. [10]

    H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al

    Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023

    2023 arXiv

  3. [12]

    K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al

    Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023 b

    2023 arXiv

  4. [13]

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope,...

    2022

  5. [14]

    Training verifiers to solve math word problems, 2021

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021

    2021

  6. [15]

    Metarm: Shifted distributions alignment via meta-learning, 2024

    Dou, S., Liu, Y., Zhou, E., Li, T., Jia, H., Xiong, L., Zhao, X., Ye, J., Zheng, R., Gui, T., Zhang, Q., and Huang, X. Metarm: Shifted distributions alignment via meta-learning, 2024. URL https://arxiv.org/abs/2405.00438

    2024 arXiv

  7. [16]

    Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024. URL https://arxiv.org/abs/2404.04475

    2024 arXiv

  8. [17]

    and Bengio, Y

    Grandvalet, Y. and Bengio, Y. Semi-supervised learning by entropy minimization. In Saul, L., Weiss, Y., and Bottou, L. (eds.), Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004. URL https://proceedings.neurips.cc/paper_files/paper/2004/file/96f2b50b...

    2004

  9. [18]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gre...

    2024 arXiv

  10. [20]

    A survey on llm-as-a-judge, 2025

    Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., Wang, Y., and Guo, J. A survey on llm-as-a-judge, 2025. URL https://arxiv.org/abs/2411.15594

    2025 arXiv

  11. [21]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations

  12. [22]

    S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J

    Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve, 2022. URL https://arxiv.org/abs/2210.11610

    2022 arXiv

  13. [23]

    Instruct2act: Mapping multi-modality instructions to robotic actions with large language model

    Huang, S., Jiang, Z., Dong, H., Qiao, Y., Gao, P., and Li, H. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305.11176, 2023

    2023 arXiv

  14. [24]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL http...

    2023 arXiv

  15. [25]

    A survey of reinforcement learning from human feedback, 2024

    Kaufmann, T., Weng, P., Bengs, V., and Hüllermeier, E. A survey of reinforcement learning from human feedback, 2024. URL https://arxiv.org/abs/2312.14925

    2024

  16. [26]

    o pf, A., Kilcher, Y., von R \

    K \"o pf, A., Kilcher, Y., von R \"u tte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D., Stanley, O., Nagyfi, R., et al. Openassistant conversations-democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024

    2024

  17. [27]

    Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N

    Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. Rewardbench: Evaluating reward models for language modeling, 2024. URL https://arxiv.org/abs/2403.13787

    2024 arXiv

  18. [28]

    Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y

    Liu, C. Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in llms, 2024. URL https://arxiv.org/abs/2410.18451

    2024 arXiv

  19. [29]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  20. [30]

    Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine

    Luo, Y., Zhang, J., Fan, S., Yang, K., Wu, Y., Qiao, M., and Nie, Z. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint arXiv:2308.09442, 2023

    2023 arXiv

  21. [31]

    Introducing ChatGPT

    OpenAI. Introducing ChatGPT . https://openai.com/blog/chatgpt, 2022

    2022

  22. [32]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 0 27730--27744, 2022

    2022

  23. [34]

    Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J

    Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization, 2024 b . URL https://arxiv.org/abs/2404.19733

    2024 arXiv

  24. [35]

    Disentangling length from quality in direct preference optimization

    Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentangling length from quality in direct preference optimization. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 4998--5017, Bangkok, Thailand, Aug...

    2024 doi

  25. [36]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  26. [37]

    Proximal policy optimization algorithms, 2017

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  27. [38]

    Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback

    Shen, W., Zheng, R., Zhan, W., Zhao, J., Dou, S., Gui, T., Zhang, Q., and Huang, X.-J. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 2859--2873, 2023

    2023

  28. [39]

    A long way to go: Investigating length correlations in rlhf

    Singhal, P., Goyal, T., Xu, J., and Durrett, G. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023

    2023 arXiv

  29. [40]

    Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  30. [41]

    Llama: Open and efficient foundation language models, 2023

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023

    2023

  31. [42]

    Aligning large language models with human: A survey, 2023

    Wang, Y., Zhong, W., Li, L., Mi, F., Zeng, X., Huang, W., Shang, L., Jiang, X., and Liu, Q. Aligning large language models with human: A survey, 2023. URL https://arxiv.org/abs/2307.12966

    2023 arXiv

  32. [43]

    Cream: Consistency regularized self-rewarding language models, 2024

    Wang, Z., He, W., Liang, Z., Zhang, X., Bansal, C., Wei, Y., Zhang, W., and Yao, H. Cream: Consistency regularized self-rewarding language models, 2024. URL https://arxiv.org/abs/2410.12735

    2024 arXiv

  33. [44]

    Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge, 2024

    Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J., Weston, J., and Sukhbaatar, S. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge, 2024. URL https://arxiv.org/abs/2407.19594

    2024 arXiv

  34. [45]

    Unsupervised data augmentation for consistency training

    Xie, Q., Dai, Z., Hovy, E., Luong, T., and Le, Q. Unsupervised data augmentation for consistency training. Advances in neural information processing systems, 33: 0 6256--6268, 2020

    2020

  35. [46]

    Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss, 2024

    Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss, 2024. URL https://arxiv.org/abs/2312.16682

    2024 arXiv

  36. [47]

    Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J

    Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. E. Self-rewarding language models. In Forty-first International Conference on Machine Learning

  37. [48]

    Consistency regularization for cross-lingual fine-tuning

    Zheng, B., Dong, L., Huang, S., Wang, W., Chi, Z., Singhal, S., Che, W., Liu, T., Song, X., and Wei, F. Consistency regularization for cross-lingual fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internation...

    2021

  38. [49]

    E., and Stoica, I

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM -as-a-judge with MT -bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Da...

    2023

  39. [50]

    P., Zhang, H., Gonzalez, J

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023 b . URL https://arxiv.org/abs/2306.05685

    2023 arXiv

  40. [51]

    Lima: Less is more for alignment, 2023

    Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., Zhang, S., Ghosh, G., Lewis, M., Zettlemoyer, L., and Levy, O. Lima: Less is more for alignment, 2023

    2023

  41. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.