Pith. sign in

REVIEW 5 major objections 5 minor 42 references

Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performance

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a frozen instruction-tuned 7B LLM, prompted with a one-line JSON rubric and a rank-16 LoRA adapter touching 0.8% of its parameters, can replace heavyweight trained reward models, reaching 96.2% on RewardBench and…

desk verdict Plausible idea, unverifiable numbers: the two headline results rest on unverified data separation, and the paper contradicts itself on judge scale; not ready for referees until artifacts and audits appear. read the letter →

arxiv 2506.05748 v1 pith:4LOUX63F submitted 2025-06-06 cs.LG cs.AI

Rudransh Agnihotri , Ananya Pandey This is my paper
classification cs.LGcs.AI
keywords rewardmodelingLLM-as-a-judgeLoRAonlinePPORLHFBenchGSM-8Kpreferencealignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reward modeling is normally the expensive bottleneck of RLHF, requiring huge preference datasets and separately trained networks. This paper claims that a frozen instruction-tuned LLM, steered by a one-line JSON rubric and fitted with a rank-16 LoRA adapter on 0.8% of its parameters, can serve as a complete substitute for those heavyweight critics. The judge reaches 96.2% on RewardBench, above specialized reward models of 27B to 70B parameters, and in an online PPO loop it lifts a 7B actor to 92% exact match on GSM-8K, beating a 70B DPO policy at 61.8% and removing the offline preference-tuning stage. The paper also contributes HH-Rationales, 10,000 human-annotated preference pairs, and reports that the LoRA judge's explanations score about 9/10 similarity to human rationales.

What carries the argument

The load-bearing object is the plug-and-play judge: a frozen instruction-tuned LLM given a system prompt that enforces a JSON output with five axis scores (correctness, safety, reasoning, factuality, clarity) and a rationale of at most 20 words, merged into a scalar reward by fixed weights (0.35, 0.25, 0.20, 0.15, 0.05). Six in-context demonstrations selected from the error-prone RewardBench slices supply most of the zero-to-few-shot gain, and a rank-16 LoRA adapter trained on RewardMix-10K, a mix of RewardBench-train pairs and general preference pairs, closes the residual gap. The same prompt and adapter are then fed directly into the PPO loop, so the scalar reward the policy optimizes is the judge's output rather than a separately trained reward model.

What would settle it

Compute the overlap between RewardMix-10K and the GSM-8K train and test sets using exact-match and near-duplicate similarity search; any test-solution hit undermines the 92% figure. A cleaner check is to retrain the LoRA judge on RewardMix-10K with all mathematics and code items removed and rerun the 300K-step PPO on GSM-8K — if exact match collapses well below 92%, the cross-domain transfer claim is unsupported.

Watch

Extended reading notes

Core claim

The central discovery, as the paper states it, is that the learned reward model can be replaced by a frozen instruction-tuned LLM whose evaluation behavior is fixed by a structured prompt, and that the only parameter update needed to close the gap is a tiny LoRA adapter. The resulting judge is deterministic, prompt-controllable, and interpretable: it emits five sub-scores (correctness, safety, reasoning, factuality, clarity) plus a short rationale before a scalar reward is extracted. On RewardBench the LoRA-adapted judge reaches 96.2%, surpassing the 27B-70B reward networks on the leaderboard, with the biggest gains on the safety and adversarial Chat-Hard segments. Used as the critic in online PPO on GSM-8K, it produces a 7B actor with 92% exact match after 300,000 steps, without any offline DPO phase; the paper argues this shows cross-domain transfer because the LoRA was trained on RewardMix-10K rather than GSM-8K preferences.

Load-bearing premise

The load-bearing premise is that RewardMix-10K, the LoRA critic's training set, is genuinely disjoint from GSM-8K, so the 92% exact-match result reflects transfer rather than memorized benchmark solutions; the paper asserts this separation in Section 6.1 without a deduplication audit.

Editorial extensions

If this is right

  • Dedicated reward-model training can be dropped from RLHF pipelines; alignment objectives are changed by editing the rubric line or swapping demonstrations, not by retraining a network.
  • An actor-critic system with a 7B actor and a 7B judge plus about 0.8% extra parameters can run within a single-GPU budget, replacing 27B-70B critics.
  • The offline DPO phase may be unnecessary for at least math reasoning: online PPO with the LoRA judge surpasses the 70B DPO baseline on GSM-8K (92% vs 61.8%).
  • Rewards become inspectable, since each scalar is accompanied by a human-like explanation that scores about 9/10 agreement with human rationales.
  • The same frozen judge can score long inputs up to its context window, with chunked minimum scoring beyond that, an ability traditional reward models often lack.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report a deduplication or hash check between RewardMix-10K and GSM-8K; a direct overlap audit would settle whether the 92% exact-match figure is genuine cross-domain transfer or an artifact of leaked solutions.
  • The recipe is tested on only one downstream benchmark, GSM-8K; whether the prompt-plus-LoRA critic transfers to coding, instruction following, or long-context tasks remains an open extension the paper does not run.
  • The rationale-agreement results use an independent LLM as the similarity scorer; a human study on the HH-Rationales test set would be a stronger check on the 9/10 interpretability claim.
  • If the pattern generalizes, the practical bottleneck shifts from training reward models to curating a few hundred demonstration and preference examples, which is a different and much cheaper engineering problem.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes replacing trained reward models in online RLHF with a frozen, instruction-tuned LLM prompted with a JSON rubric and optionally augmented with a rank-16 LoRA adapter. The method is evaluated on RewardBench, where the LoRA variant reportedly reaches 96.2% accuracy, and in an online PPO setup on GSM-8K, where a 7B actor reportedly reaches 92% exact match without an offline DPO phase. The paper also introduces HH-Rationales, a 10,000-pair interpretability dataset, and reports GPT-4-assessed rationale similarity scores around 9/10 for the LoRA judge versus about 5/10 for zero-shot judges. The central claims are that a small prompt plus a tiny adapter can replace 27B-70B reward models and outperform a 70B DPO policy on GSM-8K.

Significance. If the results were independently verified, the proposed approach would be practically significant: it promises large savings in compute and parameters, transparent and adjustable reward signals, and the removal of an offline preference-tuning phase. The prompt ablations in Tables 2, 4, and 6 are informative and show a clear effect of structured prompting and in-context demonstrations. However, the significance as presented is substantially undermined by evaluation circularity and unverified data separation. The headline RewardBench number is obtained after training on RewardBench train pairs and tuning scalar weights to RewardBench priorities, so it is a fitted evaluation rather than an independent prediction. The GSM-8K result relies on an asserted but undocumented separation between the critic's training distribution and the benchmark. Until these issues are resolved with proper held-out evaluation, released data, and a leakage audit, the state-of-the-art claims are not supported.

major comments (5)
  1. [Section 3.5 and Table 5] The headline RewardBench result is not an independent evaluation. The LoRA adapters are trained on RewardMix-10K, which contains 5K RewardBench train pairs, and Eq. (1) uses weights stated to "replicate the implicit priorities of RewardBench." Reporting accuracy on the RewardBench test split after training on its train split and tuning the scalar merge weights to that benchmark's rubrics constitutes a fitted evaluation. The abstract's claim of outperforming 27B-70B trained reward networks therefore needs support from a RewardBench-blind configuration or a held-out preference benchmark.
  2. [Section 6.1] The decisive GSM-8K result rests on an unsubstantiated data-separation assertion. The text states that "we confirm that the 92% exact-match result is not artificially enhanced through the retention of any of the benchmark's 8,500 solutions," but no deduplication procedure, hash check, version of the RewardBench split, or release of RewardMix-10K is provided. Because RewardMix-10K (Section 3.5) deliberately includes reasoning-heavy math and code content, overlap with GSM-8K prompts or responses cannot be ruled out. The authors must supply the audit, the data, and results across multiple independent seeds.
  3. [Sections 3.3, 3.5, and 4-6] The evaluation configuration is inconsistent. Section 3.3 specifies Qwen2.5-0.5B-Instruct as the judge, and Section 3.5 says the LoRA is integrated into a "Qwen-0.5B base," yet all reported RewardBench and GSM-8K results in Tables 1-7 use Qwen2.5-7B or Qwen3-8B. The exact backbone and adapter behind the abstract's "7B LLM" claim must be stated unambiguously, and if a 0.5B judge was used in any experiment, those results should be reported separately.
  4. [Section 3.4] The few-shot demonstrations are sampled from "RewardBench slices" without stating whether they come from the train or test split. If the demonstration pairs are drawn from the test split, the few-shot and LoRA static scores (Tables 3 and 5) are inflated by test-set exposure. The authors must confirm that the demonstrations are restricted to the train split and describe the exact sampling procedure.
  5. [Table 5 and Section 4.2] The comparison to the RewardBench leaderboard lacks details needed to support the state-of-the-art claim. The table lists only two baselines, reports a single run with no variance or confidence intervals, and does not identify the RewardBench version or the official evaluation script used. Without these details, the 96.2% figure cannot be verified against the leaderboard or compared fairly with the reported baselines.
minor comments (5)
  1. [Section 4] The RewardBench test split is described as 3,000 pairs in Section 4 but as 1,500 pairs in Section 4.2 and Table 4; the number should be reconciled.
  2. [Section 7] The subsections under Section 7 are misnumbered as 6.2 and 6.3 instead of 7.2 and 7.3, and the dataset name is spelled "HH-Rationals" in Section 7.1.
  3. [Figure 5] The y-axis label reads "Oveall Accuracy"; this typo should be corrected.
  4. [References] The Qwen2.5 technical report is cited as [14] pointing to an unrelated on-device paper, and references [18] and [27] duplicate the same RLAIF work; the reference list needs a careful pass.
  5. [Section 7.1] HH-Rationales is described as a new dataset of 10,000 re-annotated triplets, but no release link, license, or access procedure is given, which impedes reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the RewardBench result is a standard supervised train/test evaluation, and the GSM-8K claim is an empirical generalization rather than a definitional reduction.

full rationale

The paper's core claims are empirical evaluations, not derivations that reduce to their own inputs. The LoRA judge is trained on RewardMix-10K, which contains 5K RewardBench train pairs, and then evaluated on the RewardBench test split; this is ordinary supervised learning and reporting of held-out accuracy, not a prediction forced by construction. The scalar reward weights in Eq. (1) are fixed constants described as reflecting RewardBench's annotation priorities, but the judge's pairwise decisions still depend on the model's predicted subscores, so the reported accuracy is not determined by the weights alone. The GSM-8K online-RL result is a separate empirical claim about cross-domain transfer; the only way it could be circular is through data leakage, which is a correctness and reproducibility concern, not an analytical equivalence established anywhere in the manuscript. No load-bearing self-citations, imported uniqueness theorems, or ansatz-smuggling citations were found. The unverified data-separation assertions and the inconsistent judge-scale descriptions are serious validity risks, but they do not constitute circularity under the definitions required here.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on hand-picked scalar weights, a selected demo count, a clean train/test split for RewardBench, and a contamination-free separation between the judge's training data and GSM-8K. The paper also assumes GPT-4 ratings are a valid proxy for human explanation quality. None of these assumptions are supported by released artifacts.

free parameters (3)
  • Five axis weights in Eq. (1) = 0.35, 0.25, 0.20, 0.15, 0.05
    Hand-set to replicate RewardBench annotation priorities; the uniform-weight ablation costs 0.5pp, so the headline RewardBench score depends on this choice.
  • In-context demonstration count K = 6
    Selected by sweep over {0,2,4,6}; K=6 gives the best few-shot RewardBench accuracy and is used in the online PPO few-shot critic.
  • RewardMix-10K composition = 5K RewardBench train + 5K UltraFeedback
    The ratio and slice emphasis are hand-chosen to target safety and reasoning errors, with no sensitivity analysis of the 5K/5K split.
assumptions (4)
  • domain assumption RewardBench official train/test split is contamination-free; RewardBench-train pairs do not leak into the RewardBench test set used for the 96.2% number.
    Section 3.5 defines RewardMix-10K with 5K RewardBench-train pairs and Section 5 evaluates on RewardBench; no deduplication or split-version evidence is provided.
  • domain assumption GSM-8K test questions are absent from RewardMix-10K and from the judge's training distribution.
    Section 6.1 asserts this separation but no contamination audit or code is supplied, and RewardMix-10K is described as containing mathematical micro-proofs and code snippets.
  • domain assumption GPT-4 ratings of 0-10 are a valid measure of human-like rationale quality.
    Section 7 uses GPT-4 as an impartial evaluator without validating GPT-4 scores against human raters on the HH-Rationales test set.
  • ad hoc to paper The scalar reward in Eq. (1) with hand-set weights adequately matches human priorities on RewardBench.
    The weights are chosen to replicate RewardBench priorities and are ablated only against uniform weights, not derived from data or theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performance." pith.science (2026). https://pith.science/paper/4LOUX63F

@misc{pith2026250605748,
  author       = {Pith},
  title        = {Pith review of: Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4LOUX63F}},
  note         = {Machine review of arXiv:2506.05748}
}
read the original abstract

Reward-model training is the cost bottleneck in modern Reinforcement Learning Human Feedback (RLHF) pipelines, often requiring tens of billions of parameters and an offline preference-tuning phase. In the proposed method, a frozen, instruction-tuned 7B LLM is augmented with only a one line JSON rubric and a rank-16 LoRA adapter (affecting just 0.8% of the model's parameters), enabling it to serve as a complete substitute for the previously used heavyweight evaluation models. The plug-and-play judge achieves 96.2% accuracy on RewardBench, outperforming specialized reward networks ranging from 27B to 70B parameters. Additionally, it allows a 7B actor to outperform the top 70B DPO baseline, which scores 61.8%, by achieving 92% exact match accuracy on GSM-8K utilizing online PPO. Thorough ablations indicate that (i) six in context demonstrations deliver the majority of the zero-to-few-shot improvements (+2pp), and (ii) the LoRA effectively addresses the remaining disparity, particularly in the safety and adversarial Chat-Hard segments. The proposed model introduces HH-Rationales, a subset of 10,000 pairs from Anthropic HH-RLHF, to examine interpretability, accompanied by human generated justifications. GPT-4 scoring indicates that our LoRA judge attains approximately = 9/10 in similarity to human explanations, while zero-shot judges score around =5/10. These results indicate that the combination of prompt engineering and tiny LoRA produces a cost effective, transparent, and easily adjustable reward function, removing the offline phase while achieving new state-of-the-art outcomes for both static evaluation and online RLHF.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 23 canonical work pages

  1. [1]

    Each category of algorithms presents unique benefits and constraints, rendering their integrated application beneficial in real -world scenarios

    Introduction Offline alignment algorithms such as Direct Preference Optimization (DPO) [1] and online alignment algorithms like Reinforcement Learning from Human Feedback (RLHF) [2] or Guided Reward Policy Optimization (GRPO) [3] have emerged as highly effective techniques for aligning large language models (LLMs) [4], [5]. Each category of algorithms pre...

    work page

  2. [2]

    LLM-as- a-Judge

    Review of Literature This section primarily deals with a detailed literature review of Reinforcement Learning for enhancing self-improving models in Large Language Models (LLMs). 2.1 Reinforcement Learning for Improving LLMs Recent advancements in aligning Large Language Models (LLMs) predominantly utilize two complementary approaches: Offline Reinforceme...

    work page 2024

  3. [3]

    score" field in [-1, 1] and a short

    Methodology We investigate the effectiveness of pre-trained large language models (LLMs) [4], [5] in various inference settings, including zero -shot, few -shot, and instruction -driven contexts. A rigorous evaluation is conducted across multiple pretrained LLMs spanning diverse model families and different model sizes of the same family to enable a comp ...

    work page

  4. [4]

    𝑏𝑒𝑡𝑡𝑒𝑟":

    4. Experimentation This section assesses the performance of instruction-tuned LLMs functioning as zero-shot reward judges within the established RewardBench test suite. We compare four pretrained models, Qwen 2.5-7B, Qwen 3-8B, Llama 3-8B, and Llama 3.1 -8B, against the leading baselines introduced in Section 1. All experiments utilize the RewardBench tes...

    work page

  5. [5]

    be funnier

    This method makes reward behavior completely prompt-controllable and eliminates the reward- model training cost because the judge's weights remain frozen. For example, altering the alignment aim (such as "be funnier" or "prefer brevity") only requires a single line of editing rather than starting a new training cycle. Figure 1 Displays the prompt that was...

    work page

  6. [6]

    plug -and-play

    6. Online RLHF Integration & Policy Evaluation The earlier sections demonstrated that a structured -prompt judge, potentially enhanced with a small LoRA adapter, either matches or surpasses specialized reward networks in static evaluations. The unresolved issue is whether these “plug -and-play” critics maintain their effectiveness within an online reinfor...

    work page 2024

  7. [7]

    Which answer is better? Return ‘A’ or ‘B’ only

    5. Fine-tuning pre-trained instruction models for Reward modeling After the fine -tuning process of rank-16 LoRA adapters on RewardMix -10K, Qwen 2.5 -7B has attained an overall accuracy of 95.5%, exceeding the prior benchmark of 95.1%. The expanded Qwen 3-8B + LoRA achieves a new benchmark of 96.2%, representing a significant 1.1 percentage point increas...

    work page

  8. [8]

    8. Conclusion and Future Directions Methods Zero-Shot Few-Shot LoRA Rationale Similarity Score (0-10) 0 1 2 3 4 5 6 7 8 9 10 Distribution of Rationale Similarity Scores 5.2 5.5 4.8 5 7.1 7.3 6.5 6.8 9 9.2 8.5 8.7 0 2 4 6 8 10 Qwen 2.5-7B Qwen 3-8B Llama 3-8B Llama 3.1-8B Mean Rationale Similarity Score (0-10) Methods Zero-Shot Few-Shot LoRA The findings d...

    work page

Show all 42 references
  1. [9]

    A” or “B

    7. Rationale Agreement Study To ensure that the proposed plug-and-play judges not only align with human preference labels but also articulate their decisions in a manner akin to human reasoning, we also carried out a rationale- agreement experiment utilizing a novel HH -Ration...

  2. [10]

    Online and Offline Reinforcement Learning by Planning with a Learned Model,

    J. Schrittwieser, H. Thomas, A. Mandhane, M. Barekatain, I. Antonoglou, and D. Silver, “Online and Offline Reinforcement Learning by Planning with a Learned Model,” in Advances in Neural Information Processing Systems 35 (NeurIPS 2021), NeurIPS, Sep. 2021, p. 27580 to 27591. d...

    2021

  3. [11]

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model Oral,

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model Oral,” in Advances in Neural Information Processing Systems, New Orleans, USA: Curran Associates, Inc., 2023. Accessed: Ap...

    2023

  4. [12]

    A Survey of Reinforcement Learning from Human Feedback,

    T. Kaufmann, P. Weng, V . Bengs, and E. Hüllermeier, “A Survey of Reinforcement Learning from Human Feedback,” Apr. 30, 2024, arXiv: arXiv:2312.14925. doi: 10.48550/arXiv.2312.14925

    2024 doi

  5. [13]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,

    Z. Shao et al. , “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” Apr. 27, 2024, arXiv: arXiv:2402.03300. doi: 10.48550/arXiv.2402.03300

    2024 doi arXiv

  6. [14]

    Security and Privacy Challenges of Large Language Models: A Survey,

    B. C. Das, M. H. Amini, and Y . Wu, “Security and Privacy Challenges of Large Language Models: A Survey,” ACM Comput. Surv. , vol. 57, no. 6, pp. 1 –39, Jun. 2025, doi: 10.1145/3712001

    2025 doi

  7. [15]

    A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,

    L. Huang et al., “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,” ACM Trans. Inf. Syst., vol. 43, no. 2, pp. 1–55, Mar. 2025, doi: 10.1145/3703155

    2025 doi

  8. [16]

    Qwen2.5 Technical Report,

    Qwen et al., “Qwen2.5 Technical Report,” 2024, arXiv. doi: 10.48550/ARXIV .2412.15115

    2024 doi arXiv

  9. [17]

    Self-rewarding language models,

    W. Yuan et al., “Self-rewarding language models,” in Proceedings of the 41st International Conference on Machine Learning , in ICML’24, vol. 235. Vienna, Austria: JMLR.org, Jul. 2024, pp. 57905–57923

    2024

  10. [18]

    RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback,

    H. Lee et al. , “RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback,” in Proceedings of the Eleventh International Conference on Learning Representations (ICLR 2024), Vienna, Austria: ICLR / OpenReview.net, Oct. 2023. Accessed: Mar. 05, 2025. [Online]. ...

    2024

  11. [19]

    Evaluating Text -to-Visual Generation with Image -to-Text Generation,

    Z. Lin et al. , “Evaluating Text -to-Visual Generation with Image -to-Text Generation,” in Computer Vision – ECCV 2024: Proceedings of the 18th European Conference on Computer Vision, in Lecture Notes in Computer Science, vol. 15067. Cham: Springer Nature Switzerland, 2025, pp...

    2024 doi

  12. [20]

    more proficient

    propose strategies for LLMs to enhance their capabilities through self -generated feedback, removing reliance on annotated datasets or external reward signals. The concept of LLM-as-a-Judge has been extensively explored in recent literature [16], [21], [22]. These approaches t...

  13. [21]

    Training Language Models to Follow Instructions with Human Feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, and C. Wainwright, “Training Language Models to Follow Instructions with Human Feedback,” in Advances in Neural Information Processing Systems 35 (NeurIPS 2022), NeurIPS, Dec. 2022. doi: 10.5555/3600270.3602281

    2022

  14. [22]

    Reinforcement Learning Enhanced LLMs: A Survey,

    S. Wang et al. , “Reinforcement Learning Enhanced LLMs: A Survey,” Feb. 24, 2025, arXiv: arXiv:2412.10400. doi: 10.48550/arXiv.2412.10400

    2025 doi arXiv

  15. [23]

    Survey on Large Language Model -Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods,

    Y . Cao et al. , “Survey on Large Language Model -Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods,” IEEE Trans. Neural Netw. Learn. Syst., pp. 1–21, 2024, doi: 10.1109/TNNLS.2024.3497992

    2024

  16. [24]

    On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Acceleration,

    M. Xiang, R. Fernando, and B. Wang, “On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Acceleration,” Apr. 24, 2025, arXiv: arXiv:2504.17376. doi: 10.48550/arXiv.2504.17376

    2025 doi arXiv

  17. [25]

    Human -like Summarization Evaluation with ChatGPT,

    M. Gao, J. Ruan, R. Sun, X. Yin, S. Yang, and X. Wan, “Human -like Summarization Evaluation with ChatGPT,” Apr. 05, 2023, arXiv: arXiv:2304.02554. doi: 10.48550/arXiv.2304.02554

    2023 doi arXiv

  18. [26]

    Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge,

    J. Ye et al., “Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge,” Oct. 04, 2024, arXiv: arXiv:2410.02736. doi: 10.48550/arXiv.2410.02736

    2024 doi arXiv

  19. [27]

    LLM-as-a-Judge & Reward Model: What They Can and Cannot Do,

    G. Son, H. Ko, H. Lee, Y . Kim, and S. Hong, “LLM-as-a-Judge & Reward Model: What They Can and Cannot Do,” Oct. 02, 2024, arXiv: arXiv:2409.11239. doi: 10.48550/arXiv.2409.11239

    2024 doi arXiv

  20. [28]

    RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback,

    H. Lee et al., “RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback,” in Proceedings of the 41st International Conference on Machine Learning, ICML, Jul. 2024, pp. 26874–26901

    2024

  21. [29]

    Large Language Models Can Self -Improve,

    J. Huang et al., “Large Language Models Can Self -Improve,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Singapore: Association for Computational Linguistics, 2023, pp. 1051–1068. doi: 10.18653/v1/2023.emnlp-main.67

    2023 doi

  22. [30]

    Advancing Large Language Model Attribution through Self-Improving,

    L. Huang et al., “Advancing Large Language Model Attribution through Self-Improving,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA: Association for Computational Linguistics, 2024, pp. 3822–

    2024

  23. [32]

    Systematic Evaluation of LLM -as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates,

    H. Wei et al. , “Systematic Evaluation of LLM -as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates,” Mar. 30, 2025, arXiv: arXiv:2408.13006. doi: 10.48550/arXiv.2408.13006

    2025 doi arXiv

  24. [33]

    Can LLM be a Personalized Judge?,

    Y . R. Dong, T. Hu, and N. Collier, “Can LLM be a Personalized Judge?,” in Findings of the Association for Computational Linguistics: EMNLP 2024 , Miami, Florida, USA: Association for Computational Linguistics, 2024, pp. 10126 –10141. doi: 10.18653/v1/2024.findings-emnlp.592

    2024 doi

  25. [34]

    ReST-MCTS*: LLM Self- Training via Process Reward Guided Tree Search,

    D. Zhang, S. Zhoubian, Z. Hu, Y . Yue, Y . Dong, and J. Tang, “ReST-MCTS*: LLM Self- Training via Process Reward Guided Tree Search,” in Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024) , in Advances in Neural Information Processing Sy...

    2024

  26. [35]

    Self-Play Preference Optimization for Language Model Alignment,

    Y . Wu, Z. Sun, H. Yuan, K. Ji, Y . Yang, and Q. Gu, “Self-Play Preference Optimization for Language Model Alignment,” in Proceedings of the NeurIPS 2024 Workshop on Adaptive Foundation Models, in NeurIPS Workshop Proceedings. Vancouver, Canada: NeurIPS, Dec

    2024

  27. [37]

    Training language models to follow instructions with human feedback,

    L. Ouyang et al., “Training language models to follow instructions with human feedback,” in 36th Conference on Neural Information Processing Systems (NeurIPS 2022), New Orleans, LA, USA: NeurIPS. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/b...

    2022

  28. [38]

    Constitutional AI: Harmlessness from AI Feedback,

    Y . Bai et al., “Constitutional AI: Harmlessness from AI Feedback,” Dec. 15, 2022, arXiv: arXiv:2212.08073. doi: 10.48550/arXiv.2212.08073

    2022 doi arXiv

  29. [39]

    RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback,

    H. Lee et al., “RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback,” 2023, doi: 10.48550/ARXIV .2309.00267

    2023 doi arXiv

  30. [40]

    Efficient Memory Management for Large Language Model Serving with PagedAttention,

    W. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” Sep. 12, 2023, arXiv: arXiv:2309.06180. doi: 10.48550/arXiv.2309.06180

    2023 doi arXiv

  31. [41]

    RewardBench: Evaluating Reward Models for Language Modeling,

    N. Lambert et al., “RewardBench: Evaluating Reward Models for Language Modeling,” Jun. 08, 2024, arXiv: arXiv:2403.13787. doi: 10.48550/arXiv.2403.13787

    2024 doi arXiv

  32. [42]

    Iterative Reasoning Preference Optimization,

    R. Y . Pang, W. Yuan, K. Cho, H. He, S. Sukhbaatar, and J. Weston, “Iterative Reasoning Preference Optimization,” in Advances in Neural Information Processing Systems 37 (NeurIPS 2024) conference proceedings, NeurIPS

    2024

  33. [2024]

    Available: https://neurips.cc/virtual/2024/108142

    [Online]. Available: https://neurips.cc/virtual/2024/108142

    2024

  34. [3836]

    doi: 10.18653/v1/2024.emnlp-main.223

    2024 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.