Pith. sign in

REVIEW 26 cited by

SLiC-HF: Sequence Likelihood Calibration with Human Feedback

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.10425 v1 pith:VJ4I5GYP submitted 2023-05-17 cs.CL cs.AI

Yao Zhao , Rishabh Joshi , Tianqi Liu , Misha Khalman , Mohammad Saleh , Peter J. Liu This is my paper
classification cs.CLcs.AI
keywords humanfeedbackslic-hfdatamodelworkcalibrationfurthermore
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Learning from human feedback has been shown to be effective at aligning language models with human preferences. Past work has often relied on Reinforcement Learning from Human Feedback (RLHF), which optimizes the language model using reward scores assigned from a reward model trained on human preference data. In this work we show how the recently introduced Sequence Likelihood Calibration (SLiC), can also be used to effectively learn from human preferences (SLiC-HF). Furthermore, we demonstrate this can be done with human feedback data collected for a different model, similar to off-policy, offline RL data. Automatic and human evaluation experiments on the TL;DR summarization task show that SLiC-HF significantly improves supervised fine-tuning baselines. Furthermore, SLiC-HF presents a competitive alternative to the PPO RLHF implementation used in past work while being much simpler to implement, easier to tune and more computationally efficient in practice.

Discussion (0). Sign in to comment.

Forward citations

Cited by 26 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Function

    cs.LG 2025-06 conditional novelty 7.0 of 10

    ZSPO provably converges to a stationary policy using only the sign of preference feedback, without knowing the link function between preferences and rewards.

  2. Test-Time Scaling via Error Localization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.

  3. Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints

    eess.SP 2026-07 conditional novelty 6.0 of 10

    Pointwise constrained fine-tuning via sample-wise augmented Lagrangians and learned relaxations reduces tail constraint violations across safety, tool-calling, and re-ranking while preserving average task performance.

  4. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  5. Unbiased Alignment for Large Language Models with Noisy Preferences

    cs.LG 2026-07 conditional novelty 6.0 of 10

    URM and UDPO losses mathematically correct Bradley–Terry preference flip noise so LLMs can be aligned unbiasedly from noisy preference data without clean supervision.

  6. Adaptive Margin RLHF via Preference over Preferences

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Adaptive margins for DPO inferred from preference-over-preference comparisons improve alignment quality, with random sampling of comparisons working best overall.

  7. Phi-Ground Tech Report: Advancing Perception in GUI Grounding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Phi-Ground models achieve state-of-the-art click accuracy on five GUI grounding benchmarks for models under 10B parameters using a 40M-sample training recipe with text-first inputs, random-resize augmentation, uniform...

  8. DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)

    cs.LG 2025-07 conditional novelty 6.0 of 10

    DPO's connection to human choice theory generalizes: any monotone loss can pair with any choice model in a new KLST* class, so loss convexity is not normatively required.

  9. Improving Consistency in Vehicle Trajectory Prediction Through Preference Optimization

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Adapting SimPO preference optimization for multi-agent trajectory prediction lowers collision rates on Argoverse 2, Interaction, and Waymo Open Motion Dataset with minimal accuracy loss.

  10. ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Training a policy to prefer responses by optimizing only its own low-confidence (high-surprisal) tokens improves alignment over uniform token optimization in SimPO and DPO.

  11. Explicit Preference Optimization: No Need for an Implicit Reward Model

    cs.LG 2025-06 conditional novelty 6.0 of 10

    EXPO is a pair of explicit preference-optimization losses that provably avoid DPO's uniform-regularization and poor-interpolation failure modes and outperform DPO on Anthropic HH and IMDb.

  12. Debiasing Online Preference Learning via Preference Feature Preservation

    cs.LG 2025-06 conditional novelty 6.0 of 10

    PFP preserves the distribution of human preference features during online preference learning by mapping each new instruction to a feature set, converting it to system prompts, and training with DPO.

  13. Thompson Sampling in Online RLHF with General Function Approximation

    cs.LG 2025-05 reject novelty 6.0 of 10

    A model-free posterior sampling algorithm for online RLHF is shown to achieve O(sqrt(T)) regret when the completed function class has low Bellman eluder dimension.

  14. Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.

  15. Your Data Manifold is Secretly a Reward Model: Shell-LCC for Text-to-Video Generation

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    Shell-LCC models the high-quality data manifold as an isotropic shell to derive cost-free reward signals that improve realism and high-frequency details in text-to-video generation.

  16. Normalized Rewards for Preference Optimization

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A regularization term that conserves the combined length-normalized probability of chosen and rejected responses reduces likelihood displacement in DPO/SimPO, improves AlpacaEval and benchmark outcomes, and acts prima...

  17. Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraints

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    A modified DPO loss with a hinge margin improves small LLM alignment on AlpacaEval by about 2 points over the APO-zero baseline.

  18. Value-Free Policy Optimization via Reward Partitioning

    cs.LG 2025-06 conditional novelty 5.0 of 10

    RPO regresses the policy log-ratio against rewards centered by an empirical partition function estimated from same-prompt responses, removing the value network and reporting improved alignment, diversity, and lower toxicity.

  19. Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

    cs.LG 2025-06 conditional novelty 5.0 of 10

    SFT is cast as a special case of implicit reward learning, and a slower SFT learning rate improves downstream DPO/SimPO results by up to 25% relative win rate.

  20. On Monotonicity in AI Alignment

    math.ST 2025-06 conditional novelty 5.0 of 10

    Comparison-based preference learning losses of the studied form are locally pairwise monotone at stable minima, while individual-score and probability monotonicity require much stronger conditions that likely fail for...

  21. Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Preference signals in LLM alignment are concentrated in early response tokens, so models trained on data truncated to the first half perform as well as or better than those trained on full responses.

  22. Alignment-Aware Decoding

    cs.LG 2025-09 conditional novelty 4.0 of 10

    AAD decodes by greedily maximizing the per-token DPO/SFT log-likelihood ratio under a min-p plausibility filter and reports higher oracle-reward win rates than greedy, best-of-N, and emulated fine-tuning baselines.

  23. BPO: Revisiting Preference Modeling in Direct Preference Optimization

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Replacing DPO's relative reward margin with min(r_w, -α r_l) improves math benchmark accuracy by 5-10 points in this paper, though the exact gain figure is misreported and α is tuned on test data.

  24. On Symmetric Losses for Robust Policy Optimization with Noisy Preferences

    cs.LG 2025-05 reject novelty 4.0 of 10

    Symmetric losses preserve action rankings under symmetric label noise, and the paper's claim that they also handle asymmetric noise is invalid.

  25. Optimising Language Models for Downstream Tasks: A Post-Training Perspective

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A dissertation that repackages the author's previously published papers on continued pre-training, prompt tuning, and instruction modelling into a single narrative.

  26. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

Pith tools