Paper Citation Record · LEDGER

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration

As of 16 August 2026, this Paper Citation Record lists 31 of 31 outbound references and 1 inbound Pith citation observation for arXiv:2412.10616.

A citation records a reference. It does not transfer a finding from one paper to another.

pith.paper-citation-record.v1
2412.10616 v1

Coverage vector

measured 31 of 31 reference resolution

Typed states for the displayed outbound observations.

Source: paper_references, paper_reference_links, observed 2026-08-11T15:55:20.449861Z

measured 32 of 32 standing notices

One-hop event checks from named stored sources.

Source: scholarly_work_events, retraction_status_cache, observed 2026-08-15T06:32:42.880941+00:00

measured 1 of 1 inbound itemization

Pith citing papers itemized under the disclosed page cap.

Source: paper_references, paper_reference_links, observed 2026-08-16T01:00:59.114689Z

measured 0 of 1 external citation measurements

A source-named dated measurement, never combined with another source.

Source: pith, observed 2026-08-16T01:00:59.301176Z

Reference resolution

31 of 31 outbound references displayed

verified exact1
verified fuzzy5
unresolved25
parse uncertain0
malformed identifier0
metadata mismatch0

External citation measurements

No source-named external measurement is stored.

Outbound references

Observation 9be54856-dcf2-4041-92b5-06150ca4a498 · outbound

This paper cites Apply Azuma Hoeffding with its sample average overt + γ terms.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Apply Azuma Hoeffding with its sample average overt + γ terms

Reference 1

Resolution: verified fuzzy; raw_fallback, observed 2026-08-11T15:55:20.781548Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-15T06:32:42.880941+00:00.

Observation 45a7588e-81aa-4a12-8963-8613679a9eff · outbound

This paper cites For completeness, we first describe the preference model below: BTL-based Pairwise Preference (Dueling) Model:Consider a decision spaceD.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration For completeness, we first describe the preference model below: BTL-based Pairwise Preference (Dueling) Model:Consider a decision spaceD

Reference 2

Resolution: verified fuzzy; raw_fallback, observed 2026-08-11T15:55:20.771727Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-15T06:32:42.880941+00:00.

Observation 997853da-6905-415f-8ce0-a13a3c224fb1 · outbound

This paper cites Dataset Reset Policy Optimization for RLHF.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Dataset Reset Policy Optimization for RLHF

Reference 4

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.344726Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 4952c080-e234-4766-866f-e4d7774a6aab · outbound

This paper cites KTO: Model Alignment as Prospect Theoretic Optimization.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration KTO: Model Alignment as Prospect Theoretic Optimization

Reference 6

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.353607Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 3d883342-ed52-4d67-8bd6-d059d691205a · outbound

This paper cites Foundations of Reinforcement Learning and Interactive Decision Making.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Foundations of Reinforcement Learning and Interactive Decision Making

Reference 7

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.358418Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation d906bcd4-5162-4f85-a9e9-caef7db45874 · outbound

This paper cites Direct Language Model Alignment from Online AI Feedback.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Direct Language Model Alignment from Online AI Feedback

Reference 9

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.366546Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation f16a6eec-9670-4ec8-95b9-68c29bf7a740 · outbound

This paper cites SimPO: Simple Preference Optimization with a Reference-Free Reward.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration SimPO: Simple Preference Optimization with a Reference-Free Reward

Reference 11

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.375425Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 76aac503-48f1-477c-b8d0-d529221e938e · outbound

This paper cites Nash Learning from Human Feedback.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Nash Learning from Human Feedback

Reference 12

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.379126Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 6587a5a8-a90c-488f-9a01-024d1a61ee8c · outbound

This paper cites From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

Reference 14

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.386884Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 063a77eb-5e87-4049-8d7b-eb8c0558ba4b · outbound

This paper cites Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

Reference 15

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.390981Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 0d8f4596-48c9-487e-98d3-a8d74250d44d · outbound

This paper cites The Importance of Online Data: Understanding Preference Fine-tuning via Coverage.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration The Importance of Online Data: Understanding Preference Fine-tuning via Coverage

Reference 17

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.398995Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 8e75d090-37fc-4dae-b31b-168ccefad6f8 · outbound

This paper cites Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPs.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPs

Reference 18

Resolution: verified exact; local_arxiv, observed 2026-08-11T15:55:20.567068Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-15T06:32:42.880941+00:00.

Observation f3d36ecf-ffc9-459b-9912-f7762ac9c58b · outbound

This paper cites Is RLHF More Difficult than Standard RL?.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Is RLHF More Difficult than Standard RL?

Reference 19

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.406641Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation ad795cf1-619e-4876-ba73-0065c3d40990 · outbound

This paper cites Making RL with Preference-based Feedback Efficient via Randomization.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Making RL with Preference-based Feedback Efficient via Randomization

Reference 20

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.409826Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 99ac4f1f-8235-43d8-9c22-2d2365db660b · outbound

This paper cites Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

Reference 21

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.412841Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 99af4eb9-24cf-440a-b6ac-f0f5bb14a70d · outbound

This paper cites Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss

Reference 22

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.415875Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation f56f94a3-d0d3-4a03-a3a8-816047500123 · outbound

This paper cites Online Iterative Reinforcement Learning from Human Feedback with General Preference Model.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Online Iterative Reinforcement Learning from Human Feedback with General Preference Model

Reference 23

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.419053Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 162e90e5-149f-4d8d-aa28-12054b696c37 · outbound

This paper cites SLiC-HF: Sequence Likelihood Calibration with Human Feedback.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration SLiC-HF: Sequence Likelihood Calibration with Human Feedback

Reference 24

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.422799Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 14a9f5f7-209c-4749-896e-6db03eacc56c · outbound

This paper cites GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond

Reference 25

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.426332Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 88e81838-4d92-4525-9c47-3872bc995401 · outbound

This paper cites Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF

Reference 26

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.430135Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 32e87d64-117b-4fec-88c9-6ca08592d1df · outbound

This paper cites However due to the hybrid nature of our algorithm, we are able to derive concentration lemmas with a faster convergence rate.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration However due to the hybrid nature of our algorithm, we are able to derive concentration lemmas with a faster convergence rate

Reference 27

Resolution: verified fuzzy; raw_fallback, observed 2026-08-11T15:55:20.790917Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-15T06:32:42.880941+00:00.

Observation 5c1c55aa-9177-48aa-8dbd-32857511b00c · outbound

This paper cites We will use the abbreviation linB for this feedback model.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration We will use the abbreviation linB for this feedback model

Reference 30

Resolution: verified fuzzy; raw_fallback, observed 2026-08-11T15:55:20.760787Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-15T06:32:42.880941+00:00.

Observation 845c7a81-4884-4e38-99ce-ab8bb0da3083 · outbound

This paper cites Interested readers are encouraged to go over the proof of Lemma 9 of Saha (2021) to see the proof of lemma 4 above.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Interested readers are encouraged to go over the proof of Lemma 9 of Saha (2021) to see the proof of lemma 4 above

Reference 31

Resolution: verified fuzzy; raw_fallback, observed 2026-08-11T15:55:20.749858Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-15T06:32:42.880941+00:00.

Observation 5dd43ec3-19c7-4bde-8508-5212ae2fb123 · outbound

This paper cites Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF

Reference 1952

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.340571Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation b6545e01-b8f3-42fa-a2d0-b9a8683d8c68 · outbound

This paper cites Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences

Reference 2012

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.382821Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation c6ffbf91-c158-4f97-8d6d-24765c3f4be9 · outbound

This paper cites Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization

Reference 2017

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.349314Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 5ef03d1e-af95-4195-b97e-9944ce9e8842 · outbound

This paper cites Harnessing Density Ratios for Online Reinforcement Learning.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Harnessing Density Ratios for Online Reinforcement Learning

Reference 2019

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.333579Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation c47d63c8-2b0c-490d-9c4d-c6be1bf33edb · outbound

This paper cites Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient

Reference 2021

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.394661Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 648433e8-5327-4257-bc40-6a092fb608d9 · outbound

This paper cites Statistical Rejection Sampling Improves Preference Optimization.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Statistical Rejection Sampling Improves Preference Optimization

Reference 2022

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.371083Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 557f6430-e525-4608-ac5b-88bd1e4649fd · outbound

This paper cites REBEL: Reinforcement Learning via Regressing Relative Rewards.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration REBEL: Reinforcement Learning via Regressing Relative Rewards

Reference 2023

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.362579Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation b4eef0a0-1051-4805-ad27-5df50376f04b · outbound

This paper cites Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Reference 2024

Resolution: unresolved; no resolver link, observed 2026-08-11T15:55:20.337568Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Pith citing papers

Observation 99eb5fc0-a8a6-4d90-b54b-f04130792ae2 · inbound

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning cites this paper.

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration

Reference 2020

Resolution: metadata mismatch; local_arxiv, observed 2026-08-16T01:00:59.305430Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-15T06:32:42.880941+00:00.