pith. machine review for the scientific record. sign in

arxiv: 1909.08593 · v2 · submitted 2019-09-18 · 💻 cs.CL · cs.LG· stat.ML

Recognition: 3 theorem links

· Lean Theorem

Fine-Tuning Language Models from Human Preferences

Alec Radford, Daniel M. Ziegler, Dario Amodei, Geoffrey Irving, Jeffrey Wu, Nisan Stiennon, Paul Christiano, Tom B. Brown

Pith reviewed 2026-05-10 20:54 UTC · model grok-4.3

classification 💻 cs.CL cs.LGstat.ML
keywords language modelshuman preferencesreward learningreinforcement learningfine-tuningsummarizationtext generationpreference modeling
0
0 comments X

The pith

Language models can be fine-tuned via reinforcement learning on reward signals learned from human preference comparisons.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how to apply reward learning to language by collecting human judgments on pairs of model outputs, training a reward model from those judgments, and then using the reward model to guide reinforcement learning updates to a pretrained language model. For continuing text with a target style such as positive sentiment, the method produces good results after only 5,000 human comparisons. For summarization on the TL;DR and CNN/Daily Mail datasets, the resulting models extract whole sentences from the source while skipping preamble, which yields reasonable automatic scores and high human ratings.

Core claim

By training a reward model on human pairwise comparisons of language-model outputs and then applying reinforcement learning with that reward model, pretrained language models can be fine-tuned to continue text in desired styles or to produce summaries that focus on relevant content from long documents.

What carries the argument

A reward model trained on human pairwise comparisons of model outputs, which supplies the scalar reward signal used by proximal policy optimization to update the language model parameters.

If this is right

  • Stylistic continuation tasks reach good performance with only a few thousand human comparisons.
  • Summarization models learn to select and copy key sentences while discarding introductory material.
  • Reward learning from preferences succeeds on real language tasks where hand-crafted rewards are difficult to define.
  • The same pipeline can be reused for other tasks in which quality is best judged by humans rather than automatic metrics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach may require additional safeguards if labelers consistently favor easy-to-detect patterns that do not reflect deeper quality.
  • Scaling the number of comparisons or selecting them more efficiently could reduce the influence of any single heuristic in the learned reward model.
  • The method provides a concrete route for aligning language models to subjective criteria across domains beyond the four tasks tested.
  • Models trained this way might still need periodic re-training as human preferences shift over time or across populations.

Load-bearing premise

That human preference labels supply a consistent and generalizable measure of output quality rather than simply rewarding superficial patterns such as sentence length or verbatim copying.

What would settle it

If a model trained on the collected human preferences produces lower-quality outputs than a simple rule-based baseline (such as always copying the first few sentences) when both are evaluated by new human raters on held-out data, the claim that preferences provide a robust training signal would fail.

read the original abstract

Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper claims that reward learning from human preference comparisons can be used to fine-tune pre-trained language models on natural language tasks. It reports good performance on stylistic text continuation using only 5,000 human comparisons and, for summarization on TL;DR and CNN/Daily Mail, reasonable ROUGE scores plus strong human ratings with 60,000 comparisons, where models copy full sentences from the source while skipping preamble; the authors note this may exploit labeler heuristics rather than demonstrate genuine summarization skill.

Significance. If the results can be shown to reflect genuine preference-based learning rather than heuristic imitation, the work would be significant as an early demonstration that modest human feedback data can steer generative language models toward desired behaviors in open-ended tasks, supporting the broader goal of aligning language models with human values via RL.

major comments (2)
  1. [Abstract] Abstract: the central claim that the method yields 'very good performance' on summarization is immediately qualified by the observation that models copy whole sentences from the input (omitting preamble) and that this 'may be exploiting the fact that labelers rely on simple heuristics.' If labelers reward sentence copying, the 60k comparisons do not establish that the reward model learns summarization skill; this directly undermines the paper's assertion that the approach works for complex language tasks.
  2. [Evaluation] Evaluation sections: no error bars, confidence intervals, or statistical tests are reported for the human judgments or ROUGE scores, and the manuscript provides insufficient detail on the exact protocol for collecting the 5k/60k comparisons or on how the reward model is trained and applied in RL fine-tuning. These omissions make it impossible to evaluate the reliability or reproducibility of the reported gains.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below, agreeing where revisions are warranted to improve clarity and rigor.

read point-by-point responses

  1. Referee: The abstract claims 'very good performance' on summarization but qualifies it by noting sentence copying that may exploit labeler heuristics, undermining the claim that the approach works for complex tasks.

    Authors: We agree the abstract phrasing risks overstating the summarization results. The observed behavior demonstrates that the reward model successfully captures human preferences (leading to high human ratings and reasonable ROUGE), but as noted in the paper, this may rely on heuristics rather than deep summarization skill. We will revise the abstract to remove the unqualified 'very good performance' claim, explicitly state the copying behavior, and clarify that the results validate preference-based steering even when preferences align with simple heuristics. revision: yes

  2. Referee: No error bars or statistical tests for human judgments or ROUGE; insufficient details on comparison collection protocol, reward model training, and RL fine-tuning.

    Authors: We acknowledge these omissions reduce reproducibility. In revision we will add error bars and confidence intervals to all reported human evaluation and ROUGE results, include statistical significance tests where appropriate, and expand the methods sections with precise protocols for collecting the 5k/60k comparisons, reward model training details (including architecture, loss, and hyperparameters), and the exact RL fine-tuning procedure (PPO settings, KL coefficient, etc.). revision: yes

Circularity Check

0 steps flagged

No circularity: empirical pipeline grounded in independent human evaluations

full rationale

The paper's core contribution is an empirical pipeline: collect human preference comparisons, train a reward model on them, then apply RL (with KL penalty) to fine-tune a pretrained LM. Results on stylistic continuation and summarization are reported via separate human labelers and ROUGE scores. No derivation, equation, or 'prediction' reduces to the training data by construction; the method does not rename a fit as a forecast or import uniqueness via self-citation chains. The paper itself notes the summarization heuristic risk, treating it as an empirical observation rather than a definitional loop. The derivation chain is therefore self-contained against external human judgments.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The work rests on the assumption that human pairwise preferences can be modeled as a scalar reward function that generalizes beyond the collected comparisons. No new physical entities or mathematical axioms beyond standard RL and supervised learning are introduced.

free parameters (1)
  • number of human comparisons
    5,000 for stylistic tasks and 60,000 for summarization are chosen quantities that determine reported performance.
axioms (1)
  • domain assumption Human preferences over model outputs can be captured by a learned reward model that generalizes to new generations.
    Invoked when training the reward model from comparisons and then optimizing the policy against it.

pith-pipeline@v0.9.0 · 5492 in / 1308 out tokens · 56913 ms · 2026-05-10T20:54:19.472350+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Efficient Preference Poisoning Attack on Offline RLHF

    cs.LG 2026-05 unverdicted novelty 8.0

    Label-flip attacks on log-linear DPO reduce to binary sparse approximation problems that can be solved efficiently by lattice-based and binary matching pursuit methods with recovery guarantees.

  2. Language Models are Few-Shot Learners

    cs.CL 2020-05 accept novelty 8.0

    GPT-3 shows that scaling an autoregressive language model to 175 billion parameters enables strong few-shot performance across diverse NLP tasks via in-context prompting without fine-tuning.

  3. Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment

    cs.LG 2026-05 unverdicted novelty 7.0

    Temperature adjustment on the reference model generalizes inference-time alignment to SLOP ensembles of reward models, with a calibration algorithm that improves robustness to reward hacking while preserving alignment...

  4. Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models

    cs.LG 2026-05 unverdicted novelty 7.0

    Reinforce Adjoint Matching derives a simple consistency loss for RL post-training of diffusion models by tilting the clean distribution toward higher-reward samples under KL regularization while keeping the noising pr...

  5. Structure from Strategic Interaction & Uncertainty: Risk Sensitive Games for Robust Preference Learning

    cs.GT 2026-05 unverdicted novelty 7.0

    Risk-sensitive preference games retain monotonicity via translation-invariant risk measures, enabling convergent self-play algorithms with stability bounds and empirical robustness across data strata.

  6. BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models

    cs.AI 2026-05 unverdicted novelty 7.0

    BoostAPR improves automated program repair by using execution-grounded RL with a sequence-level assessor and line-level credit allocator, reaching 40.7% on SWE-bench Verified and strong cross-language results.

  7. Convex Optimization with Nested Evolving Feasible Sets

    cs.LG 2026-05 unverdicted novelty 7.0

    For convex losses in nested evolving feasible sets, a lazy algorithm balances O(T^{1-β}) regret with O(T^β) movement for any β; for strongly convex or sharp losses, Frugal achieves zero regret with O(log T) movement, ...

  8. Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs

    cs.CL 2026-05 unverdicted novelty 7.0

    RL on binary rewards boosts LLM factual recall by ~27% relative across models by redistributing probability mass to latent correct answers rather than acquiring new knowledge.

  9. $f$-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses

    cs.LG 2026-05 unverdicted novelty 7.0

    The paper establishes the first O(log T) regret and O(1/T) sub-optimality bounds for online RLHF under general f-divergence regularization via two sampling algorithms.

  10. Towards Robust LLM Post-Training: Automatic Failure Management for Reinforcement Fine-Tuning

    cs.SE 2026-05 unverdicted novelty 7.0

    Introduces the first benchmark for fine-grained failures in reinforcement fine-tuning of LLMs and an automatic management framework that detects, diagnoses, and remediates them.

  11. Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards

    cs.AI 2026-05 unverdicted novelty 7.0

    TraceLift trains reasoning planners with executor-grounded rewards that multiply a rubric-based reasoning quality score by measured uplift on a frozen executor, outperforming execution-only training on math and code b...

  12. Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent

    cs.LG 2026-05 unverdicted novelty 7.0

    Reference-sampled weighted SFT with prompt-normalized Boltzmann weights induces the same policy as fixed-reference KL-regularized RLVR, with BOLT as the estimator and a finite one-shot error decomposition separating c...

  13. Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare

    cs.LG 2026-05 unverdicted novelty 7.0

    The work establishes a regret lower bound of Ω(T^{2/3} min(K,D)^{1/3}) for fair multi-user dueling bandits with heterogeneous Condorcet winners and gives algorithms achieving matching upper bounds up to logs.

  14. Three Models of RLHF Annotation: Extension, Evidence, and Authority

    cs.CY 2026-04 unverdicted novelty 7.0

    RLHF should decompose annotations into dimensions each matched to one of three models—extension, evidence, or authority—instead of applying a single unified pipeline.

  15. Interactive Episodic Memory with User Feedback

    cs.CV 2026-04 unverdicted novelty 7.0

    Introduces an interactive episodic memory task with user feedback and a Feedback Alignment Module that improves retrieval accuracy on video benchmarks while remaining efficient.

  16. Reverse Constitutional AI: A Framework for Controllable Toxic Data Generation via Probability-Clamped RLAIF

    cs.CL 2026-04 unverdicted novelty 7.0

    R-CAI inverts constitutional AI to automatically generate diverse toxic data for LLM red teaming, with probability clamping improving output coherence by 15% while preserving adversarial strength.

  17. Teaching LLMs Human-Like Editing of Inappropriate Argumentation via Reinforcement Learning

    cs.CL 2026-04 unverdicted novelty 7.0

    Reinforcement learning with a multi-part reward teaches LLMs to output independent, meaning-preserving sentence edits that raise argument appropriateness close to full rewriting.

  18. E2E-REME: Towards End-to-End Microservices Auto-Remediation via Experience-Simulation Reinforcement Fine-Tuning

    cs.SE 2026-04 unverdicted novelty 7.0

    E2E-REME outperforms nine LLMs in accuracy and efficiency for end-to-end microservice remediation by using experience-simulation reinforcement fine-tuning on a new benchmark called MicroRemed.

  19. From OSS to Open Source AI: an Exploratory Study of Collaborative Development Paradigm Divergence

    cs.SE 2026-04 conditional novelty 7.0

    Open source AI shows lower collaboration intensity, reduced direct contributions, and a shift toward adaptive use rather than joint improvement compared to traditional OSS.

  20. Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedback

    cs.LG 2026-03 unverdicted novelty 7.0

    Introduces robust estimators for linear Markov games in offline MARLHF that achieve O(ε^{1-o(1)}) or O(√ε) bounds on Nash or CCE gaps under uniform or unilateral coverage.

  21. Group-in-Group Policy Optimization for LLM Agent Training

    cs.LG 2025-05 unverdicted novelty 7.0

    GiGPO adds a hierarchical grouping mechanism to group-based RL so that LLM agents receive both global trajectory and local step-level credit signals, yielding >12% gains on ALFWorld and >9% on WebShop over GRPO while ...

  22. KTO: Model Alignment as Prospect Theoretic Optimization

    cs.LG 2024-02 conditional novelty 7.0

    KTO aligns LLMs by directly maximizing prospect-theoretic utility on binary signals and matches or exceeds preference-based methods like DPO from 1B to 30B parameters.

  23. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

    cs.LG 2024-01 conditional novelty 7.0

    Medusa augments LLMs with multiple decoding heads and tree-based attention to predict and verify several tokens in parallel, yielding 2.2-3.6x inference speedup via two fine-tuning regimes.

  24. Self-Rewarding Language Models

    cs.CL 2024-01 conditional novelty 7.0

    Iterative self-rewarding via LLM-as-Judge in DPO training on Llama 2 70B improves instruction following and self-evaluation, outperforming GPT-4 on AlpacaEval 2.0.

  25. Measuring Faithfulness in Chain-of-Thought Reasoning

    cs.AI 2023-07 conditional novelty 7.0

    Chain-of-Thought reasoning in LLMs is often unfaithful, with models relying on it variably by task and less so as models scale larger.

  26. Let's Verify Step by Step

    cs.LG 2023-05 accept novelty 7.0

    Process supervision significantly outperforms outcome supervision for training models on the MATH dataset, achieving 78% accuracy on a representative test subset with active learning and a released 800k step-label dataset.

  27. Red Teaming Language Models with Language Models

    cs.CL 2022-02 conditional novelty 7.0

    One language model can generate diverse test cases to automatically uncover tens of thousands of harmful behaviors, including offensive replies and privacy leaks, in a large target language model.

  28. Driving Intents Amplify Planning-Oriented Reinforcement Learning

    cs.RO 2026-05 unverdicted novelty 6.0

    DIAL uses intent-conditioned CFG and multi-intent GRPO to expand and preserve diverse modes in continuous-action preference RL, lifting RFS to 9.14 and surpassing both prior best (8.5) and human demonstration (8.13).

  29. Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training

    cs.LG 2026-05 unverdicted novelty 6.0

    A new RL objective adapts trust-region and off-policy handling automatically via normalized effective sample size of batch policy ratios, matching tuned baselines without new hyperparameters.

  30. Discrete Flow Matching for Offline-to-Online Reinforcement Learning

    cs.LG 2026-05 unverdicted novelty 6.0

    DRIFT enables stable offline-to-online fine-tuning of CTMC policies in discrete RL via advantage-weighted discrete flow matching, path-space regularization, and candidate-set approximation.

  31. PriorZero: Bridging Language Priors and World Models for Decision Making

    cs.LG 2026-05 unverdicted novelty 6.0

    PriorZero uses root-only LLM prior injection in MCTS and alternating world-model training with LLM fine-tuning to raise exploration efficiency and final performance on Jericho text games and BabyAI gridworlds.

  32. TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

    cs.CL 2026-05 unverdicted novelty 6.0

    TBPO derives a token-level preference optimization objective from sequence-level pairwise data via Bregman divergence ratio matching that generalizes DPO and improves alignment quality.

  33. Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

    cs.LG 2026-05 unverdicted novelty 6.0

    Standard preference learning induces spurious feature reliance via mean bias and correlation leakage, creating irreducible distribution shift vulnerabilities that tie training mitigates without degrading causal learning.

  34. Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping

    cs.CV 2026-05 unverdicted novelty 6.0

    Super-Linear Advantage Shaping (SLAS) introduces a non-linear geometric policy update for RL post-training of text-to-image models that reshapes the local policy space via advantage-dependent Fisher-Rao weighting to r...

  35. Guaranteed Jailbreaking Defense via Disrupt-and-Rectify Smoothing

    cs.CR 2026-05 unverdicted novelty 6.0

    DR-Smoothing introduces a disrupt-then-rectify prompt processing scheme into smoothing defenses, delivering tight theoretical bounds on success probability against both token- and prompt-level jailbreaks.

  36. Annotations Mitigate Post-Training Mode Collapse

    cs.CL 2026-05 unverdicted novelty 6.0

    Annotation-anchored training reduces semantic diversity collapse in post-trained language models by a factor of six compared to standard supervised fine-tuning while preserving instruction-following and improving with scale.

  37. BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models

    cs.AI 2026-05 unverdicted novelty 6.0

    BoostAPR boosts automated program repair by training a sequence-level assessor and line-level credit allocator from execution outcomes, then applying them in PPO to reach 40.7% on SWE-bench Verified.

  38. BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models

    cs.AI 2026-05 unverdicted novelty 6.0

    BoostAPR uses supervised fine-tuning on verified fixes, dual sequence- and line-level reward models from execution feedback, and PPO to reach 40.7% on SWE-bench Verified with strong cross-language results.

  39. $\xi$-DPO: Direct Preference Optimization via Ratio Reward Margin

    cs.LG 2026-05 unverdicted novelty 6.0

    ξ-DPO rewrites the preference objective as minimizing distance to optimal margins and defines reward as a chosen-to-rejected ratio, yielding a bounded, interpretable margin ξ set directly from the initial reward-gap d...

  40. Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph

    cs.LG 2026-05 unverdicted novelty 6.0

    GraphDPO generalizes pairwise DPO to a graph-structured Plackett-Luce objective over DAGs induced by rollout rankings, enforcing transitivity with linear complexity and recovering DPO as a special case.

  41. SOD: Step-wise On-policy Distillation for Small Language Model Agents

    cs.CL 2026-05 unverdicted novelty 6.0

    SOD reweights on-policy distillation strength step-by-step using divergence to stabilize tool use in small language model agents, yielding up to 20.86% gains and 26.13% on AIME 2025 for a 0.6B model.

  42. Implicit Preference Alignment for Human Image Animation

    cs.CV 2026-05 unverdicted novelty 6.0

    IPA aligns animation models for superior hand quality via implicit reward maximization on self-generated samples plus hand-focused local optimization, avoiding expensive paired data.

  43. Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models

    cs.LG 2026-05 unverdicted novelty 6.0

    Mutual Reinforcement Learning allows heterogeneous LLMs to exchange experience through mechanisms like Peer Rollout Pooling, Cross-Policy GRPO Advantage Sharing, and Success-Gated Transfer, with outcome-level sharing ...

  44. Response Time Enhances Alignment with Heterogeneous Preferences

    cs.LG 2026-05 unverdicted novelty 6.0

    Response times modeled as drift-diffusion processes enable consistent estimation of population-average preferences from heterogeneous anonymous binary choices.

  45. On the Blessing of Pre-training in Weak-to-Strong Generalization

    cs.LG 2026-05 unverdicted novelty 6.0

    Pre-training provides a geometric warm start in a single-index model that enables weak-to-strong generalization up to a supervisor-limited bound, with empirical phase-transition evidence in LLMs.

  46. Stayin' Aligned Over Time: Towards Longitudinal Human-LLM Alignment via Contextual Reflection and Privacy-Preserving Behavioral Data

    cs.HC 2026-05 unverdicted novelty 6.0

    A methodological framework and browser system BITE for collecting evolving user preferences on LLM outputs through context-triggered reflections and privacy-preserving data over time.

  47. Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards

    cs.AI 2026-05 unverdicted novelty 6.0

    TraceLift trains reasoning planners using rewards that credit traces for both rubric quality and actual performance gains on a frozen executor, outperforming final-answer-only training on math and code tasks.

  48. Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models

    cs.AI 2026-05 unverdicted novelty 6.0

    MoR lets clients train local reward models on private preferences and uses a learned Mixture-of-Rewards with GRPO on the server to align a shared base VLM without exchanging parameters, architectures, or raw data.

  49. DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment

    cs.LG 2026-05 unverdicted novelty 6.0

    DGPO reinterprets distribution deviation as a guiding signal in a critic-free policy optimization framework to enable fine-grained credit assignment for LLM chain-of-thought reasoning.

  50. DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment

    cs.LG 2026-05 unverdicted novelty 6.0

    DGPO is a critic-free RL framework that uses bounded Hellinger distance and entropy-gated advantage redistribution to enable fine-grained token-level credit assignment in long CoT generations for LLM alignment, report...

  51. Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models

    cs.LG 2026-05 conditional novelty 6.0

    Gate-DPO attenuates gradients on low-probability rejected responses to reduce probability collapse and improve chosen-response likelihood during preference optimization.

  52. Binary Rewards and Reinforcement Learning: Fundamental Challenges

    cs.LG 2026-05 unverdicted novelty 6.0

    Binary rewards make the set of reward-maximizing policies infinite in policy gradients; KL control selects the filtered base model but misspecification drives collapse to concentrated valid outputs instead.

  53. A Meta Reinforcement Learning Approach to Goals-Based Wealth Management

    cs.LG 2026-05 unverdicted novelty 6.0

    MetaRL pre-trained on GBWM problems delivers near-optimal dynamic strategies in 0.01s achieving 97.8% of DP optimal utility and handles larger problems where DP fails.

  54. Minimizing Collateral Damage in Activation Steering

    cs.LG 2026-05 unverdicted novelty 6.0

    Activation steering is cast as constrained optimization that minimizes collateral damage by weighting perturbations according to the empirical second-moment matrix of activations instead of assuming isotropy.

  55. Diversity in Large Language Models under Supervised Fine-Tuning

    cs.LG 2026-04 unverdicted novelty 6.0

    TOFU loss mitigates the narrowing of generative diversity in LLMs after supervised fine-tuning by addressing neglect of low-frequency patterns and forgetting of prior knowledge.

  56. Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback

    cs.LG 2026-04 unverdicted novelty 6.0

    DRRO for RLHF replaces worst-case value with worst-case regret in Wasserstein DRO, producing an exact water-filling solution under l1 ambiguity and a practical sampled-bonus algorithm that reduces proxy over-optimization.

  57. Test-Time Safety Alignment

    cs.CL 2026-04 unverdicted novelty 6.0

    Optimizing input embeddings sub-lexically via black-box zeroth-order gradients neutralizes all safety-flagged responses from aligned models on standard benchmarks.

  58. When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient

    cs.LG 2026-04 unverdicted novelty 6.0

    Certain errors in proxy rewards for policy gradient methods can be benign or beneficial by preventing policies from stalling on outputs with mediocre ground truth rewards, enabling improved RLHF metrics and reward des...

  59. MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment

    cs.LG 2026-04 unverdicted novelty 6.0

    MGDA-Decoupled applies geometry-based multi-objective optimization within the DPO framework to find shared descent directions that account for each objective's convergence dynamics, yielding higher win rates on UltraFeedback.

  60. AlignCultura: Towards Culturally Aligned Large Language Models?

    cs.CL 2026-04 unverdicted novelty 6.0

    Align-Cultura introduces the CULTURAX dataset and shows that culturally fine-tuned LLMs improve joint HHH scores by 4-6%, cut cultural failures by 18%, and gain 10-12% efficiency with minimal leakage.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · cited by 81 Pith papers · 5 internal anchors

  1. [1]

    Deep batch active learning by diverse, uncertain gradient lower bounds.arXiv preprint arXiv:1906.03671, 2019

    Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch ac- tive learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671,

    work page arXiv 1906

  2. [2]

    Learning to understand goal specifications by mod- elling reward

    Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefen- stette. Learning to understand goal specifications by mod- elling reward. arXiv preprint arXiv:1806.01946,

    work page arXiv

  3. [3]

    Supervising strong learners by amplifying weak experts

    Paul Christiano, Buck Shlegeris, and Dario Amodei. Super- vising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575,

    work page Pith review arXiv

  4. [4]

    Preference-based interactive multi-document summarisa- tion

    Yang Gao, Christian M Meyer, and Iryna Gurevych. Preference-based interactive multi-document summarisa- tion. arXiv preprint arXiv:1906.02923, 2019a. Yang Gao, Christian M Meyer, Mohsen Mesgar, and Iryna Gurevych. Reward learning for efficient reinforcement learning in extractive document summarisation. arXiv preprint arXiv:1907.12894, 2019b. Sebastian Geh...

    work page arXiv 1906

  5. [5]

    Discriminative active learning

    Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv preprint arXiv:1907.06347,

    work page arXiv 1907

  6. [6]

    Learning from dialogue af- ter deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415,

    Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. Learning from dialogue af- ter deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415,

    work page arXiv 1901

  7. [7]

    Universal language model fine-tuning for text classification

    Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification.arXiv preprint arXiv:1801.06146,

    work page arXiv

  8. [8]

    Active learning for speech recognition: the power of gradients

    Jiaji Huang, Rewon Child, Vinay Rao, Hairong Liu, Sanjeev Satheesh, and Adam Coates. Active learning for speech recognition: the power of gradients. arXiv preprint arXiv:1612.03226,

    work page arXiv

  9. [9]

    Reward learning from human preferences and demonstrations in atari, 2018, 1811.06521 http://arxiv.org/abs/1811.06521

    URL https://arxiv.org/abs/1811.06521. Geoffrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899,

    work page arXiv

  10. [10]

    AI safety via debate

    URL https://arxiv.org/abs/1805.00899. Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In Pro- ceedings of the 34th International Conference on Ma- chine Learning-Volume 70, pages 1645–1654. JMLR. org,

    work page internal anchor Pith review arXiv

  11. [11]

    Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.arXiv preprint arXiv:1907.00456,

    Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456,

    work page arXiv 1907

  12. [12]

    Sample efficient text summarization using a single pre-trained transformer

    Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser. Sample efficient text summarization using a single pre-trained transformer. arXiv preprint arXiv:1905.08836,

    work page arXiv 1905

  13. [13]

    Adam: A Method for Stochastic Optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Reli- ability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning

    Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reli- ability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. arXiv preprint arXiv:1805.10627,

    work page arXiv

  15. [15]

    Neural text summarization: A critical evaluation

    Wojciech Kry´sci´nski, Nitish Shirish Keskar, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. Neural text summarization: A critical evaluation. arXiv preprint arXiv:1908.08960,

    work page arXiv 1908

  16. [16]

    Scalable agent alignment via reward modeling: a research direction

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent align- ment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871,

    work page Pith review arXiv

  17. [17]

    Dialogue learning with human-in-the-loop

    Jiwei Li, Alexander H Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston. Dialogue learning with human-in-the-loop. arXiv preprint arXiv:1611.09823 ,

    work page arXiv

  18. [18]

    Reinforcement learning for bandit neural machine trans- lation with simulated human feedback

    Fine-Tuning Language Models from Human Preferences Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement learning for bandit neural machine trans- lation with simulated human feedback. arXiv preprint arXiv:1707.07402,

    work page arXiv

  19. [19]

    A deep reinforced model for abstractive summarization

    Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304,

    work page arXiv

  20. [20]

    Finding gener- alizable evidence by learning to convince Q&A models

    Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason We- ston, Douwe Kiela, and Kyunghyun Cho. Finding gener- alizable evidence by learning to convince Q&A models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China, November

    work page 2019

  21. [21]

    Deep contextualized word representations

    Association for Computational Linguistics. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gard- ner, Christopher Clark, Kenton Lee, and Luke Zettle- moyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365,

    work page Pith review arXiv

  22. [22]

    Learning to Generate Reviews and Discovering Sentiment , publisher =

    Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learn- ing to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444,

    work page arXiv

  23. [23]

    1511.06732 , archiveprefix =

    URL https://d4mucfpksywv.cloudfront. net/better-language-models/language_ models_are_unsupervised_multitask_ learners.pdf. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732,

    work page arXiv

  24. [24]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    Get To The Point: Summarization with Pointer-Generator Networks

    Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator net- works. arXiv preprint arXiv:1704.04368,

    work page Pith review arXiv

  26. [26]

    Neural Machine Translation of Rare Words with Subword Units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909,

    work page internal anchor Pith review arXiv

  27. [27]

    Controllable neural story generation via reinforcement learning

    Pradyumna Tambwekar, Murtaza Dhuliawala, Animesh Mehta, Lara J Martin, Brent Harrison, and Mark O Riedl. Controllable neural story generation via reinforcement learning. arXiv preprint arXiv:1809.10736,

    work page arXiv

  28. [28]

    Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mo- hammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap be- tween human and machine translation. arXiv preprint arXiv:1609.08144,

    work page internal anchor Pith review arXiv

  29. [29]

    Towards coherent and engaging spoken dialog response generation us- ing automatic conversation evaluators

    Sanghyun Yi, Rahul Goel, Chandra Khatri, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur. Towards coherent and engaging spoken dialog response generation us- ing automatic conversation evaluators. arXiv preprint arXiv:1904.13015,

    work page arXiv 1904