REVIEW 12 cited by
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs can improve by judging their own responses instead of relying on human labelers. However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills. Surprisingly, this unsupervised approach improves the model's ability to judge {\em and} follow instructions, as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard. These results strongly suggest the potential for self-improving models without human supervision.
Forward citations
Cited by 12 Pith papers
-
Multilingual Self-Taught Faithfulness Evaluators
STEMF trains multilingual faithfulness evaluators from synthetic data alone, and English-only training yields the best average results across languages.
-
SGPO: Self-Generated Preference Optimization based on Self-Improver
SGPO uses one shared model to refine its own responses and then optimize with DPO on those self-generated preference pairs, outperforming DPO and SPIN on AlpacaEval 2.0 and Arena-Hard without external preference labels.
-
Bridging Offline and Online Reinforcement Learning for LLMs
Semi-online DPO, which syncs the generation model every few update steps, performs nearly as well as fully online DPO and GRPO, while strongly beating offline DPO.
-
From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
ProxyReward trains long-form generation models by rewarding how well an AI judge can answer generated yes/no questions about the response, improving open-source models on ProxyQA.
-
PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier
A new multi-turn reinforcement learning framework trains a single LLM to both solve math problems and verify its own solutions, revising only when its verifier finds a mistake.
-
Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
CoVo trains LLMs with a self-generated reward based on the consistency and volatility of intermediate reasoning states, matching supervised RL performance without external labels.
-
Unlocking Recursive Thinking of LLMs: Alignment via Refinement
An offline alignment pipeline using reward-filtered self-refinement data and long chain-of-thought SFT raises an 8B model's AlpacaEval 2 win rate from 25.0% to 51.0% with roughly 14k training examples.
-
Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solution
LLM-as-a-Judge systems report confidence that overstates their accuracy, and the paper's TH-Score plus LLM-as-a-Fuser improves calibration.
-
Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
Anchoring rejected responses to the initial model and choosing responses from a future model raises AlpacaEval 2.0 win rate from 19.69 to 29.44 for Llama3.1-8B.
-
Enterprise Large Language Model Evaluation Benchmark
A 14-task enterprise LLM benchmark built mostly from GPT-4o-generated labels and scored by GPT-4o-as-judge shows open-source models closing the reasoning gap, but the dataset is not public and the evaluation is partly...
-
Revisiting Active Learning under (Human) Label Variation
A position paper that surveys and systematizes how active learning should change when human label variation is treated as a signal rather than noise.
- Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models
Discussion (0). Sign in to comment.