Pith. sign in

REVIEW 4 cited by

Improving LLM General Preference Alignment via Optimistic Online Mirror Descent

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.16852 v1 pith:JECZIIMD submitted 2025-02-24 cs.LG cs.AIcs.CL

Yuheng Zhang , Dian Yu , Tao Ge , Linfeng Song , Zhichen Zeng , Haitao Mi , Nan Jiang , Dong Yu This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords alignmentassumptionhumanpreferencesdescentgeneralimprovinglearning
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Reinforcement learning from human feedback (RLHF) has demonstrated remarkable effectiveness in aligning large language models (LLMs) with human preferences. Many existing alignment approaches rely on the Bradley-Terry (BT) model assumption, which assumes the existence of a ground-truth reward for each prompt-response pair. However, this assumption can be overly restrictive when modeling complex human preferences. In this paper, we drop the BT model assumption and study LLM alignment under general preferences, formulated as a two-player game. Drawing on theoretical insights from learning in games, we integrate optimistic online mirror descent into our alignment framework to approximate the Nash policy. Theoretically, we demonstrate that our approach achieves an $O(T^{-1})$ bound on the duality gap, improving upon the previous $O(T^{-1/2})$ result. More importantly, we implement our method and show through experiments that it outperforms state-of-the-art RLHF algorithms across multiple representative benchmarks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Safety Alignment of LMs via Non-cooperative Games

    cs.AI 2025-12 conditional novelty 7.0 of 10

    Jointly training an Attacker and Defender LLM in a non-zero-sum game with pairwise preference judges produces a defender with much lower jailbreak success while preserving general utility.

  2. Supervised Fine-Tuning vs. In-Context Learning: An Equilibrium Analysis of LLM Personalization under Congestion

    cs.LG 2026-07 conditional novelty 6.0 of 10

    In a linear model of LLM personalization with shared compute, SFT beats ICL above a coverage-dependent signal-to-noise threshold, congestion can reverse that ranking, and adding SFT never reduces platform profit.

  3. Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    LLM self-play finetuning is equivalent to adversarial imitation learning; the chi-squared regularized variant SPIF bounds rewards and improves stability.

  4. SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence

    cs.CL 2025-02 conditional novelty 6.0 of 10

    SelfElicit uses deep-layer attention to automatically highlight relevant evidence sentences in the input context, yielding consistent QA accuracy gains across six instruction-tuned LLMs.

Pith tools