Pith. sign in

REVIEW 5 major objections 7 minor 31 references

Correcting Large Language Model Behavior via Influence Function

T0 review · 5 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that LANCET, a two-phase influence-function method, can correct undesirable LLM behavior without human intervention by tracing bad outputs to influential training samples and re-optimizing on their influence ranking.

desk verdict LANCET is a solid empirical contribution to automatic LLM behavior correction, but the influence-score approximation needs direct validation before the causal mechanism is credible. read the letter →

arxiv 2412.16451 v1 pith:TMNAWJR4 submitted 2024-12-21 cs.LG cs.AIcs.CL

Han Zhang , Zhuo Zhang , Yi Zhang , Yuanzhao Zhai , Hanyang Peng , Yu Lei , Yue Yu , Hui Wang
show 3 more authors
Bin Liang Lin Gui Ruifeng Xu
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords influencefunctionsLLMbehaviorcorrectionmodelunlearningdirectpreferenceoptimizationGauss-NewtonHessianKronecker-factoredapproximationsafealignmentout-of-distributiongeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to correct behaviors a large language model learned from outdated or inappropriate training data without collecting fresh human feedback. It claims that influence functions can locate the training examples most responsible for a given undesired output, and that a new post-training objective can then reshape the model so those outputs become unlikely while its general utility is preserved. If true, model behavior could be kept in step with changing social norms at the cost of computing influence scores rather than paying annotators. The paper reports that its pipeline outperforms both human-correction alignment baselines and gradient-ascent unlearning methods, especially on out-of-distribution harmful prompts.

What carries the argument

The central object is the influence score of a training sample on an influence query, computed through the Proximal Bregman Response Function. LinFAC makes it tractable for LLMs by treating each Transformer sublayer as one linearized module, summing over token activations and pre-activation pseudo-gradients so token dependencies are kept, and approximating the modular Gauss-Newton Hessian as a Kronecker product of sequence-summed factors $\hat{A}\otimes\hat{S}$. The forward pass is carried by the inverse-Hessian-vector product $\hat{G}^{-1}(\sum_t a_t^{z_q}\otimes D_s^{z_q}_t)$, which reduces to small matrix solves. IBO then converts the ranked scores into a pairwise correction loss, a Bregman-divergence forgetfulness penalty, and a proximity regularizer.

What would settle it

Train a small transformer on contaminated data, delete each of the top-100 recalled training examples one at a time, retrain, and measure the change in the harmful query's probability; if the rank correlation between true retraining effects and LinFAC scores is near zero, the recall stage supplies wrong data and IBO's correction signal is built on noise.

Watch

Extended reading notes

Core claim

LANCET treats behavior correction as two linked tasks. First, LinFAC scores every recalled training sample by how much it increases or decreases the probability of the unwanted response, using a linearized Transformer sublayer and a Kronecker-factored approximation of the Gauss-Newton Hessian. Second, Influence-driven Bregman Optimization (IBO) turns those scores into a pairwise ranking objective, much like direct preference optimization but with influence rankings replacing human preference labels, and adds a Bregman-divergence regularizer on non-influential samples so the model does not forget its other knowledge. The paper reports that this pipeline lowers harmful responses on seen and unseen prompts across Safe RLHF, BeaverTails, and Anthropic-HH, with smaller utility loss than gradient-ascent unlearning and stronger out-of-distribution generalization than SFT or DPO retrained on human-corrected data.

Load-bearing premise

Everything depends on the assumption that LinFAC's approximate influence scores rank training examples in the same order as actually deleting each example and retraining the model would; the paper gives precision-recall curves against known unsafe labels but never directly checks this ranking against real retraining.

Editorial extensions

If this is right

  • If LinFAC rankings are faithful, harmful behavior can be traced to specific training examples without manual annotation, and corrected by editing model behavior rather than retraining from scratch.
  • IBO's pairwise influence-ranking objective can be trained with samples from different prompts, so correction signals are not limited to same-prompt preference pairs.
  • The approach should plug into any LLM whose raw training set is accessible, including continual-alignment pipelines where old preference data becomes outdated.
  • Batch-query IHVP with clustered influence queries reduces computation and improves correction by averaging out cross-harmful-behavior noise, suggesting influence estimation should be done over sets of queries rather than single prompts.
  • According to the paper, LANCET lowers harmfulness on out-of-distribution prompts more than methods that collect human-corrected data, implying influence-based correction transfers beyond the observed bad outputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ranking assumption holds, the same machinery could serve as a training-data audit: influence scores would flag stale, poisoned, or otherwise harmful examples even before a specific bad output is observed.
  • The case study's pairing of top-1 and last-1 samples suggests correction may be driven by semantic opposition of instructions; a direct test would compare IBO against a variant that pairs samples with opposite labels but shuffles influence magnitudes.
  • One could test a cheaper deployment: use LinFAC only for recall and replace IBO with standard DPO on synthesized pairs; if correction holds, the expensive Hessian machinery is only needed for data selection, not for training.
  • The paper's safety-focused benchmarks leave open whether the same method corrects non-safety preference drift, such as style or factual recency, which is a natural next experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes LANCET, a two-stage method to correct undesirable LLM behavior without human annotation. In the first stage, the new LinFAC influence-function estimator recalls training examples that increase or decrease the probability of undesirable outputs (influence queries). In the second stage, Influence-driven Bregman Optimization (IBO) trains the model with a pairwise loss over positively and negatively influential samples, plus a Bregman divergence term on non-influential samples to preserve utility. The authors evaluate on Safe RLHF, BeaverTails, and Anthropic-HH with Llama 3.1, Llama 2, and OPT backbones, reporting reduced harmfulness on seen and unseen prompts while maintaining helpfulness, and they provide efficiency, compatibility, and case-study analyses.

Significance. If the influence scores are reliable, LANCET offers a plausible human-free alternative to preference-data collection and model unlearning, with the additional appeal of making the recalled training samples interpretable. The paper's strengths include the modular two-stage design, evaluation across three datasets and three backbones, a compatibility matrix (Table 7), explicit wall-clock comparisons (Figure 4e), and a concrete case study (Table 8). The central risk is that both the recall stage and the correction signal depend on an unvalidated Kronecker-factor approximation, and the evaluation pipeline reuses the same reward/cost models for query selection, checkpoint selection, and reporting. These issues are addressable with additional validation and ablations.

major comments (5)
  1. [Section 3.1, Appendix A.2, Eqs. (9)-(13), (20)-(22)] The load-bearing premise of the method is that LinFAC influence scores rank training samples by their causal effect on the query output distribution. This premise is not validated. Eq. (20) replaces the sequence-level sum of Kronecker products \sum_t a_t \otimes Ds_t with a single Kronecker product (1/T \sum_t a_t) \otimes (\sum_t Ds_t), and Eq. (21) then replaces an expectation of an outer product by a product of expectations. Neither step is an equality, and no error bound or small-scale comparison against leave-one-out retraining is provided. The PR curves in Figure 4(a-c) only measure recovery of injected unsafe samples under a coarse classification metric, so they do not establish that the scores reflect the true effect on p(z_r|z_p). Because DIF_+ and DIF_- in Eq. (15) are built directly from these scores, contaminated rankings can propagate into the IBO loss and the interpretability claims. I would like to see (i) a comparison of LinFAC rankings with exact leave-one-out influence or full-Hessian influence on a small model, reported by rank correlation or NDCG, and (ii) an ablation that replaces LinFAC with random or TF-IDF recall inside IBO to quantify how much of the behavioral improvement comes from the influence ranking.
  2. [Section 4.1, Appendix D.2, Appendix D.3] The evaluation protocol shares the same reward/cost model signal across query selection, checkpoint selection, and final evaluation. Section 4.1 selects influence queries by the largest increase in harmfulness according to the cost model; Appendix D.2 selects the best post-training checkpoint by "the maximal RM-CM score on a valid set"; and Appendix D.3 computes the reported Helpful/Harmless scores with the same family of RM/CM models. This creates a selection-on-the-evaluation-metric risk: the reported gains, especially on unseen prompts, may be inflated by optimizing the same scores used for measurement. Please evaluate with a held-out set of human ratings on unseen prompts, or with a different reward/cost model that was not used for selection, and state explicitly that the validation set used for checkpointing is disjoint from the test set.
  3. [Section 3.2, Appendix D.2] Several hyperparameters that directly control the correction strength are not reported. The final objective in Eq. (17) contains the proximity regularizer weight \lambda; Eq. (15) contains \epsilon and the Pareto threshold \alpha; Eq. (13) has the batch/cluster size K; and the recall pipeline has a TF-IDF recall size. The text states only that "we set \epsilon = -1" and that the model is trained for 4000 steps. Without the values of \lambda, \alpha, and K, and without a sensitivity analysis for \lambda and \alpha, the method cannot be reproduced and the robustness of the reported gains cannot be assessed. Please report all hyperparameter values and include a sensitivity study for at least \lambda and \alpha.
  4. [Section 1, Section 4.1] The motivating problem is correcting behaviors caused by outdated or erroneous human preferences, but the experiments operationalize "inappropriate" exclusively as harmful or unsafe content. The paper acknowledges this in Section 4.1 ("no existing data is available to simulate this scenario"), yet the abstract and introduction claim corrections that make LLMs deviate from contemporary human preferences and societal norms. Harmfulness is a reasonable proxy, but it is not the same as preference drift: a preference shift need not involve safety, and the mechanism of learning new preferences differs from suppressing unsafe responses. I recommend either adding a preference-drift experiment (for example, flipping chosen/rejected pairs in a preference dataset over time) or explicitly narrowing the claims in the abstract and conclusion to unsafe-behavior correction until such evidence exists.
  5. [Section 3.2 Eq. (15), Table 7] On the seen influence queries, the correction is partly by construction. Eq. (15) optimizes a pairwise loss constructed from the influence of each recalled training sample on the query log-probability, so reducing the probability of those exact queries is the designed effect and should not be reported as an independent discovery. The paper does evaluate unseen prompts, which is the right test, but the current ablations do not isolate the contribution of the influence ranking. Table 7 varies the recall strategy (LinFAC vs. EK-FAC) and the post-training objective (IBO vs. PBO), but it does not include a random-recall or TF-IDF-only baseline with IBO. Please add such an ablation, and report the human Likert or a held-out metric on unseen prompts for it; this is necessary to support the claim that influence-function recall, rather than fine-tuning on recalled harmful data, drives the generalization gains.
minor comments (7)
  1. [Throughout] There are typos and naming inconsistencies, including "an novel" in the abstract, "Bregman Diversity" in Appendix A.1, "deviding" in Section 3.2, and inconsistent "Linear-FAC"/"LinFAC" usage; please proofread the manuscript.
  2. [Appendix A.2, Eq. (21)] The normalization in Eq. (21) seems inconsistent: the left factor carries an extra 1/N while Eq. (22) assigns 1/N to \hat S; please make the factor derivation internally consistent.
  3. [Section 4.4, Figure 4(f)] The plot lacks axis labels, and the text describes "positive intervention (epsilon=+1)" and "negative intervention (epsilon=-1)", but Eq. (15) sets epsilon=-1 in the loss; the sign convention should be clarified.
  4. [Figure 4(a-c)] The PR curves do not state the retrieval cutoff or the number of candidate samples used to compute precision and recall; please specify these details so the comparison with EK-FAC is reproducible.
  5. [Appendix D.1] Anthropic-HH is cited as (Ji et al. 2024); the correct source appears to be Bai et al. (2022a); please correct the citation.
  6. [Table 6] The statement that batch query improves performance is not uniformly supported by the table, since the IF-Query Harmless score is higher (worse) for LANCET_B (0.616) than for LANCET_S (0.581); please clarify which metrics support the claim.
  7. [Tables 4 and 5] No significance tests are reported; given the overlapping standard deviations in several rows, please add statistical tests or confidence intervals for the headline comparisons.

Circularity Check

1 steps flagged · score 6.0 of 10

Seen-query behavior correction is constructed from the influence scores on those same queries; unseen generalization provides independent support.

  1. fitted input called prediction [Section 3.2, Eqs. (14)-(15); Section 4.2, Table 3 'IF Query' column]
    "According to the definition of influence score, a positive/negative influence score indicates that the training sample increases/decreases the likelihood of generating undesirable behavior. Based on this, correcting the undesirable behavior can be considered as: maxθ Ez∼DIF[−If(z,zq,θ) − β log πθ(y|x)/πθs(y|x)]. ... Our method considers the influence ranking as the preference and constructs pairwise data to learn the optimal solution for objective Eq. 14."

    The influence score If(z,zq) in Eqs. (6)/(9) is, by definition, the derivative of the undesirable response's log-probability log p(zr|zp) with respect to upweighting the training sample z. Eq. 14 then defines the correction objective as maximizing −If(z,zq,θ), i.e., directly lowering that same probability for the influence query. The DIF+ and DIF− sets and the If weights used in the pairwise loss Eq. 15 are selected and scaled from the same query-specific influence scores. Consequently, the reported decrease in the 'IF Query Harmful' column of Table 3 is the objective being optimized, not an independent prediction about those queries.

full rationale

The paper's derivation chain is otherwise largely self-contained. LinFAC is a Kronecker-factored approximation of the influence-function IHVP, and the recall stage is validated against known injected unsafe samples via PR curves, which is an external ground truth rather than a circular target. The generalization results on unseen prompts and utility preservation are measured on held-out data and do not reuse the influence scores as training targets, so the strongest transfer claims are not circular. The one clear circular element is the seen-query correction: the influence scores are computed with respect to the same Influence Queries on which 'IF Query Harmful' is later reported, and Eq. 14 states that correction means maximizing the negative of those very influence scores. Thus the improvement on seen queries is the designed objective, not a discovered prediction. Appendix C's admission that the IBO pairs 'seem to differ somewhat from collecting human preferences' further tempers the interpretability claim, but that is a support/correctness concern rather than a circularity. No load-bearing self-citation chain was found.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The method relies on a chain of approximations: influence-function theory, K-FAC factorization, a new sequence-sum Kronecker approximation, TF-IDF preselection, and an external reward/cost model. Several key hyperparameters are unreported. No fundamentally new entities are postulated.

free parameters (6)
  • lambda (damping and proximity regularizer)
    Used in PBRF (Eq. 4) and the IBO objective (Eq. 17); value not reported in the paper, yet it controls the strength of the correction and the regularization.
  • alpha (Pareto selection threshold)
    Section 3.2 uses alpha to define D+ and D-; the paper says alpha follows a Pareto distribution but gives no concrete value or selection procedure.
  • K (number of IF query clusters for batch IHVP) = 10
    Section 4.4 and Figure 5: K-Means with k=10 is used to group influence queries; no sensitivity analysis.
  • TF-IDF recall size = 100
    Appendix D.2: 'We first use TF-IDF to recall 100 samples and rank them by influence score.' This hard cap can exclude influential samples.
  • epsilon in the IBO pairwise loss = -1
    Section 3.2: set to -1 to flip the sign of the ranking loss; the reason is heuristic and not derived.
  • Number of post-training steps and validation checkpoint selection = 4000 steps; max RM-CM on valid set
    Appendix D.2: the best checkpoint is chosen by maximizing the reward-minus-cost score on a validation set, which uses the same evaluation signal as the main results.
assumptions (7)
  • domain assumption Influence functions computed via the proximal Bregman response function approximate the true effect of removing a training sample on model behavior.
    Section 2 Eqs. 4-6 adopts Bae et al. (2022) PBRF; the paper relies on this approximation being accurate enough for LLMs despite known bias of standard influence functions.
  • standard math The Gauss-Newton Hessian equals the Fisher information matrix, and activations are approximately independent of pre-activation gradients so the Hessian factorizes as A ⊗ S.
    Section 3.1 Eq. 7 and Appendix A.2; standard K-FAC-style factorization assumption, but the independence of activations and gradients is an approximation.
  • ad hoc to paper The sum over tokens of Kronecker products can be approximated by the Kronecker product of token sums (Eq. 20 in Appendix A.2).
    This is a new approximation introduced for LinFAC; no error bound is given, and it is central to the modular GNH estimate.
  • domain assumption TF-IDF retrieval of the top 100 training samples recovers all samples that meaningfully influence the undesirable behavior.
    Section 3.1 Step 3 and Appendix D.2; if an influential sample is lexically dissimilar to the query, it is never scored.
  • domain assumption Unsafe or harmful examples in the training set are a valid proxy for 'outdated or erroneous' human preferences.
    Section 4.1 and Figure 3; the paper tests behavior correction on safety, but the motivating scenario is changing societal norms; the transfer is asserted, not measured.
  • domain assumption Open-source reward and cost models provide accurate measures of helpfulness and harmfulness.
    Section 4.1 and D.3; all model-based scores and checkpoint selection depend on these external classifiers.
  • domain assumption Correcting the probability of a small set of influence queries is sufficient to reduce undesirable behavior on unseen prompts.
    Section 2 Problem Statement and Section 4.3; the method trains only on samples related to known influence queries and relies on generalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Correcting Large Language Model Behavior via Influence Function." pith.science (2026). https://pith.science/paper/TMNAWJR4

@misc{pith2026241216451,
  author       = {Pith},
  title        = {Pith review of: Correcting Large Language Model Behavior via Influence Function},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TMNAWJR4}},
  note         = {Machine review of arXiv:2412.16451}
}
read the original abstract

Recent advancements in AI alignment techniques have significantly improved the alignment of large language models (LLMs) with static human preferences. However, the dynamic nature of human preferences can render some prior training data outdated or even erroneous, ultimately causing LLMs to deviate from contemporary human preferences and societal norms. Existing methodologies, whether they involve the curation of new data for continual alignment or the manual correction of outdated data for re-alignment, demand costly human resources. To address this challenge, we propose a novel approach, Large Language Model Behavior Correction with Influence Function Recall and Post-Training (LANCET), which requires no human involvement. LANCET consists of two phases: (1) using influence functions to identify the training data that significantly impact undesirable model outputs, and (2) applying an Influence function-driven Bregman Optimization (IBO) technique to adjust the model's behavior based on these influence distributions. Our experiments demonstrate that LANCET effectively and efficiently correct inappropriate behaviors of LLMs. Furthermore, LANCET can outperform methods that rely on collecting human preferences, and it enhances the interpretability of learning human preferences within LLMs.

Figures

Figures reproduced from arXiv: 2412.16451 by the authors.

Figure 1
Figure 1. The Framework of AI Behavior Correction. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of LANCET. LANCET first uses Lin￾FAC and IQs to identify training samples that cause (positive influence score) and inhibit (negative influence scores) LLM misbehavior. These IF-scored samples are subsequently em￾ployed for behavior correction by IBO. Suppose that the Transformer sublayer f with parameter θ = {θi} L i=1 where θi denotes the i-th linear layer of f. Lin￾FAC modularizes f as a linear layer… view at source ↗
Figure 3
Figure 3. The inappropriate samples (unsafe data) and corre [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Efficiency and performance analysis of LinFAC [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Single IF Query v.s. Batch IF Query [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 16 canonical work pages

  1. [1]

    Bae, J.; Ng, N.; Lo, A.; Ghassemi, M.; and Grosse, R. B. 2022. If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 35: 17953--17967

    work page 2022

  2. [2]

    Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; DasSarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; Joseph, N.; Kadavath, S.; Kernion, J.; Conerly, T.; El-Showk, S.; Elhage, N.; Hatfield-Dodds, Z.; Hernandez, D.; Hume, T.; Johnston, S.; Kravec, S.; Lovitt, L.; Nanda, N.; Olsson, C.; Amodei, D.; Brown, T.; Clark, J.; McCandlish, S.; Olah, ...

    arXiv 2022

  3. [3]

    E.; Fort, S.; Lanham, T.; Telleen-Lawton, T.; Conerly, T.; Henighan, T.; Hume, T.; Bowman, S

    Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; Chen, C.; Olsson, C.; Olah, C.; Hernandez, D.; Drain, D.; Ganguli, D.; Li, D.; Tran-Johnson, E.; Perez, E.; Kerr, J.; Mueller, J.; Ladish, J.; Landau, J.; Ndousse, K.; Lukosuite, K.; Lovitt, L.; Sellitto, M.; Elhage, N.; Schiefer, N.; ...

    arXiv 2022

  4. [4]

    Brown, T. B. 2020. Language models are few-shot learners. arXiv preprint ArXiv:2005.14165

    arXiv 2020

  5. [5]

    Carroll, M.; Foote, D.; Siththaranjan, A.; Russell, S.; and Dragan, A. 2024. AI Alignment with Changing and Influenceable Reward Functions. In ICLR 2024 Workshop: How Far Are We From AGI

    work page 2024

  6. [6]

    Dai, J.; Pan, X.; Sun, R.; Ji, J.; Xu, X.; Liu, M.; Wang, Y.; and Yang, Y. 2023. Safe RLHF : Safe Reinforcement Learning from Human Feedback. arXiv:2310.12773

    arXiv 2023

  7. [7]

    Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    E.; Schneider, F.; and Hennig, P

    Eschenhagen, R.; Immer, A.; Turner, R. E.; Schneider, F.; and Hennig, P. 2023. Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures. In Thirty-seventh Conference on Neural Information Processing Systems

    work page 2023

Show all 31 references
  1. [9]

    Feldman, V.; and Zhang, C. 2020. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 2881--2891

    2020

  2. [10]

    George, T.; Laurent, C.; Bouthillier, X.; Ballas, N.; and Vincent, P. 2018. Fast approximate natural gradient descent in a kronecker factored eigenbasis. Advances in Neural Information Processing Systems, 31

    2018

  3. [11]

    Grosse, R.; Bae, J.; Anil, C.; Elhage, N.; Tamkin, A.; Tajdini, A.; Steiner, B.; Li, D.; Durmus, E.; Perez, E.; et al. 2023. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296

    2023 arXiv

  4. [12]

    Hampel, F. R. 1974. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346): 383--393

    1974

  5. [13]

    Jacot, A.; Hongler, C.; and Gabriel, F. 2018. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. In Bengio, S.; Wallach, H. M.; Larochelle, H.; Grauman, K.; Cesa - Bianchi, N.; and Garnett, R., eds., Advances in Neural Information Processing Systems 31: ...

    2018

  6. [14]

    H.; Ghandeharioun, A.; Ferguson, C.; Lapedriza, A.; Jones, N.; Gu, S.; and Picard, R

    Jaques, N.; Shen, J. H.; Ghandeharioun, A.; Ferguson, C.; Lapedriza, A.; Jones, N.; Gu, S.; and Picard, R. 2020. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),...

    2020

  7. [15]

    Ji, J.; Liu, M.; Dai, J.; Pan, X.; Zhang, C.; Bian, C.; Chen, B.; Sun, R.; Wang, Y.; and Yang, Y. 2024. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36

    2024

  8. [16]

    Kreutzer, J.; Khadivi, S.; Matusov, E.; and Riezler, S. 2018. Can Neural Machine Translation be Improved with User Feedback? In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volu...

    2018

  9. [17]

    Ortiz-Jim \'e nez, G.; Moosavi-Dezfooli, S.-M.; and Frossard, P. 2021. What can linearized neural networks actually say about generalization? Advances in Neural Information Processing Systems, 34: 8998--9010

    2021

  10. [18]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language model...

    2022

  11. [19]

    D.; Ermon, S.; and Finn, C

    Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  12. [20]

    Ramamurthy, R.; Ammanabrolu, P.; Brantley, K.; Hessel, J.; Sifa, R.; Bauckhage, C.; Hajishirzi, H.; and Choi, Y. 2022. Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

    2022

  13. [21]

    M.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; and Christiano, P

    Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D. M.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; and Christiano, P. F. 2020. Learning to summarize from human feedback. CoRR, abs/2009.01325

    2020 arXiv

  14. [22]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  15. [23]

    Yang, R.; Pan, X.; Luo, F.; Qiu, S.; Zhong, H.; Yu, D.; and Chen, J. 2024. Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment. International Conference on Machine Learning

    2024

  16. [24]

    Yao, Y.; Xu, X.; and Liu, Y. 2024. Large Language Model Unlearning. arXiv:2310.10683

    2024 arXiv

  17. [25]

    Zhang, H.; Lei, Y.; Gui, L.; Yang, M.; He, Y.; Wang, H.; and Xu, R. 2024. CPPO: Continual Learning for Reinforcement Learning with Human Feedback. In The Twelfth International Conference on Learning Representations

    2024

  18. [26]

    V.; et al

    Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  19. [27]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  20. [28]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  21. [29]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  22. [30]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  23. [31]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2000

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.