Pith. sign in

REVIEW 4 major objections 5 minor 17 references

Influence Functions for Preference Dataset Pruning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Removing the 10% of training examples that influence functions rate most harmful improves reward-model test accuracy by about 1.5 percentage points over training on the full dataset, while using 10% less data.

desk verdict A small, honest empirical note on influence pruning for preference data; the non-significant 1.5% and stochastic CG make it a conditional pass, but the helpful-vs-harmful asymmetry is worth a look. read the letter →

arxiv 2507.14344 v1 pith:LSGZKSTA submitted 2025-07-18 cs.LG cs.AI

Daniel Fein , Gabriela Aranguiz-Dias This is my paper · ORCID
classification cs.LGcs.AI
keywords influencefunctionsdatasetpruningpreferencedatasetsrewardmodelingRLHFLoRAconjugategradientdatacuration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether influence-function approximations can clean noisy human-preference datasets used to train reward models. It adapts the TL;DR dataset to a pairwise preference task, computes per-example influence in the LoRA parameter space of a 1B-parameter model, and prunes the training examples that most strongly push the reward model away from a small validation set. Removing the worst 10% of examples improves test accuracy by about 1.5 percentage points relative to full-data training, even while using less data. The paper also finds that a cheaper first-order approximation, gradient similarity, is better than influence functions at identifying helpful examples, while influence functions are better at identifying harmful ones.

What carries the argument

The central object is the classical influence score $I_{\mathrm{IF}}(z_i,z_j) = -\nabla_\theta \mathcal{L}(z_j)^\top H_\theta^{-1} \nabla_\theta \mathcal{L}(z_i)$, which estimates how much removing training example $z_i$ would change the loss on validation example $z_j$. Because the Hessian of a billion-parameter model is too large to invert, the paper restricts all gradients to the low-rank LoRA adapter subspace (about 0.12% of weights, roughly 1.2M scalars) and approximates the inverse-Hessian-vector product $H_\theta^{-1} \nabla_\theta \mathcal{L}(z_j)$ by solving a damped linear system $(H_\theta+\lambda I)x = \nabla_\theta \mathcal{L}(z_j)$ with conjugate gradients, using double-backprop for Hessian-vector products. Each training example is scored by its mean influence $\bar{I}(z_i)$ across a 100-pair validation set, and the most positive (most harmful) examples are pruned before retraining.

What would settle it

Re-run the 10% pruning experiment on the unfiltered TL;DR dataset or another human-preference dataset with a validation set of at least 1,000 pairs; if test accuracy does not exceed full-data training, the central claim fails. The paper itself notes that the 1.5% gain is not statistically significant at the current test size, so a larger evaluation is the direct test.

Watch

Extended reading notes

Core claim

The central claim is that conjugate-gradient approximated influence functions can be used to filter noisy human-preference data for reward-model training, and that this filtering yields a small but consistent accuracy uplift: removing the 10% of training examples with the most positive mean influence score improves reward-model test accuracy by about 1.5 percentage points relative to training on the full dataset. A second claim is that gradient similarity outperforms influence functions for detecting helpful training examples, which the authors interpret as evidence that local curvature is important for identifying harmful examples but less so for identifying helpful ones.

Load-bearing premise

The results rely on the 100-pair validation set and the heavily filtered 8.5k-example training subset being representative enough of human preference data that the ranking effects generalize beyond this particular slice of TL;DR.

Editorial extensions

If this is right

  • Reward-model training on noisy preference data can be improved by pruning the most positively influential examples, with the best observed gain at 10% removal.
  • Gradient similarity can identify helpful examples so effectively that removing its top 30% estimated helpful examples drives reward-model accuracy to random chance.
  • Influence-based pruning removes harmful examples more effectively than random pruning or gradient similarity, so noise-removal applications should use the Hessian-aware score.
  • The pruning pipeline operates in the LoRA adapter space rather than the full parameter space, which makes influence estimation tractable at post-training scale.
  • Pruning also brings a data-efficiency benefit: the model uses less training data without a drop in accuracy relative to full-data training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the authors' curvature interpretation is right, a cheap proxy for local curvature might replace full influence functions when the goal is only to find harmful examples, which would make the method practical on larger datasets.
  • The small validation set and heavily filtered training subset leave open whether the same ranking effects appear on unfiltered preference data; repeating the pruning with a 1,000-pair validation set would test that directly.
  • The two scores' complementary strengths suggest a combined curation recipe—gradient similarity to keep helpful examples and influence functions to delete harmful ones—that the paper does not itself evaluate.
  • Because the paper estimates that 20-40% of preference data is noisy, even a modest accuracy gain from removing 10% of examples likely understates the benefit of removing all detected noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes using conjugate-gradient (CG) approximated influence functions in the LoRA parameter subspace to prune noisy human-preference data for reward-model training. It adapts the TL;DR dataset into a heavily filtered 8.5k-pair training set, computes per-example influence scores against a 100-pair validation set, and evaluates retraining accuracy after removing the most positively influential (harmful) examples. The paper reports a 1.5% accuracy uplift over full-data training when 10% of examples are pruned, explicitly states this gain is not statistically significant, and additionally finds that gradient similarity outperforms influence functions for identifying helpful examples. The discussion speculates that local curvature is important for harmful examples but less so for helpful ones.

Significance. If the influence ranking is reliable, the paper would provide a useful demonstration of influence-function-based data curation for preference data and an interesting, testable asymmetry between helpful and harmful examples. The authors are honest about the statistical weakness of their central result, and the computational setup (LoRA parameter space, CG-based inverse-Hessian approximation) is a sensible direction for post-training-scale influence estimation. However, the central quantitative claim is not currently established: the reported gain is not statistically significant, and the CG approximation procedure has a load-bearing technical flaw that affects the reliability of the ranking. The conceptual claim about curvature would be valuable if supported by a validated influence estimate.

major comments (4)
  1. [Section 3.3, Algorithm 1 (lines 5–6)] The CG loop draws a fresh random batch of B=20 training examples for every Hessian-vector product, so the linear operator being inverted changes at each iteration. Standard CG requires a fixed symmetric positive-definite operator; this stochastic, non-stationary variant is not guaranteed to converge to H^{-1}g, and the resulting per-example influence scores may be dominated by batch-sampling variance rather than by the true inverse-Hessian-vector product. Because the pruning decision rests on the sign of the mean influence score, this approximation gap is load-bearing. The paper should either fix a single batch for all K iterations, average over multiple HVP evaluations, or otherwise verify the residual against the full-data Hessian; the iteration budget K is also not reported.
  2. [Section 4, Figure 1] The paper states that the 1.5% improvement over full-data training is not statistically significant (test set n=1000), and no repeated seeds or variance estimates across retraining runs are reported. The 95% Wald intervals shown reflect only the binomial sampling of the test set, not the variance induced by the pruning and retraining procedure. Consequently, the central claim of a pruning benefit is currently indistinguishable from noise, and the comparison between influence pruning and other methods at 10% exclusion is not statistically supported. Please report results across multiple seeds and provide a significance test or a confidence interval for the pruning effect.
  3. [Section 3.1 and Section 7] The dataset modification discards all pairs with either summary longer than 24 Llama3 tokens, removes the original article text, and retains only about 8.5k pairs, with influence scores computed against a 100-pair validation sample. The paper acknowledges limited generalizability in Section 7, but the abstract and conclusion frame the result as a demonstration on human preference data. At minimum, the representativeness of this filtered slice should be quantified (for example, by comparing test-set accuracy and preference-noise statistics against the unfiltered TL;DR reward-model setup), and the sensitivity of the pruning ranking to the 24-token threshold and to the 100-pair validation sample should be reported.
  4. [Section 3.3 (Tikhonov damping)] The damping parameter λ=1e-2 is set without sensitivity analysis. Since λ directly controls the conditioning of the approximate inverse-Hessian, it can affect the sign of influence scores for near-flat directions and therefore determines which examples are pruned. The paper should report whether the pruning ranking and the retraining accuracy are stable over a range of λ (for example, 1e-3 to 1e-1). Without this, the qualitative distinction between helpful and harmful examples in Section 5 is not robustly tied to the curvature hypothesis.
minor comments (5)
  1. [Abstract and Section 4] The abstract describes a 'small retraining accuracy uplift of 1.5%' without noting that the paper itself reports this gain is not statistically significant; please add the qualifier to avoid overstatement.
  2. [Section 3.3, text before Algorithm 1] The sentence 'To computing an HVP without forming Hθ' contains a typo; it should read 'To compute an HVP...'.
  3. [Algorithm 1] The symbol B is used both for the batch size and for the sampled batch in line 5, which is confusing; please use distinct names, e.g., m for batch size and S for the sampled subset.
  4. [Figure 1] The panel titles 'Excluding Most Helpful Examples' and 'Excluding Most Harmful Examples' are ambiguous because the left panel removes helpful examples and the right panel removes harmful ones; please clarify in the caption what each panel shows.
  5. [References] The reference list contains both Bai et al. 2022a and Bai et al. 2022b with identical titles and URLs; please verify whether these are distinct papers and format accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: influence scores are computed from a full-data model and evaluated on a disjoint test split; no target quantity is used to fit the scores.

full rationale

The claimed derivation chain is empirical rather than definitional: a LoRA reward model is trained on the full filtered TL;DR dataset; influence scores are estimated via CG-approximated inverse-Hessian-vector products against a 100-pair validation sample; the most positively influential examples are pruned; the model is retrained from the original checkpoint with identical hyperparameters; and performance is measured on the untouched remainder of the validation split. No parameter (lambda=1e-2, batch size 20, CG budget, pruning percentage) is fitted to the reported test accuracy, and the validation sample used for influence computation is disjoint from the test set by construction. The comparison against gradient similarity and random baselines is an empirical evaluation, not a consequence of the influence formula. There are no load-bearing self-citations: the references to Koh & Liang (2017), Grosse et al. (2023), and Martens et al. (2010) are standard external prior work, and the authors do not cite any of their own previous results. The CG solver's use of a fresh random batch in each iteration is a possible correctness or robustness limitation, but it does not make the target H^{-1}g equivalent to the approximation; the target is defined independently. The paper's own admission that the 1.5% gain is not statistically significant is an evidentiary weakness, not circularity. No step in the pipeline reduces, by the paper's own equations or by self-citation, to its own inputs.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper contributes no new mathematical entities or fitted constants beyond standard hyperparameters. The main free choices are the damping parameter, validation-sample size, and an aggressive token-length filter that transforms the TL;DR dataset. The key axioms are the standard influence-function Taylor approximation and the assumption that the modified dataset and 100-pair validation set are representative.

free parameters (6)
  • Tikhonov damping lambda = 1e-2
    Chosen by hand to make the Hessian positive definite for CG (Section 3.3); not tuned on the test set, but affects the inverse-Hessian-vector product and therefore the influence scores.
  • Validation sample size |Dval| = 100
    Uniformly sampled validation pairs for computing influence scores (Section 3.1); a larger sample would change rankings.
  • Token-length filter threshold = 24 tokens (Llama-3 tokenizer)
    All examples where both summaries exceed 24 tokens are removed to fit the computation budget (Section 3.1); this heavily changes the dataset.
  • LoRA rank and alpha = r=8, alpha=16
    Standard LoRA hyperparameters (Section 3.2); they define the parameter space over which influence is computed.
  • CG mini-batch size B = 20
    Batch size for the batched Hessian-vector product (Algorithm 1); introduces stochasticity into the influence approximation.
  • CG iteration budget K = not reported
    The number of conjugate-gradient iterations is required in Algorithm 1 but never specified, making the approximation irreproducible.
assumptions (5)
  • standard math Influence functions approximate the change in validation loss from removing a training example via the classical Taylor expansion.
    Relies on Koh and Liang (2017); the inverse-Hessian-vector product is used as the influence score.
  • domain assumption The Bradley-Terry pairwise model is an appropriate loss for reward modeling.
    The reward model is trained to assign higher reward to the chosen summary (Section 3.1).
  • domain assumption LoRA parameter space captures the relevant training dynamics for attribution.
    Only LoRA weights (0.12% of parameters) are used for gradient and Hessian computations (Section 3.3).
  • ad hoc to paper The 100-pair validation sample and the filtered 8.5k-pair training subset are representative of human preference data.
    The dataset is modified by token-length filtering and post-text removal (Section 3.1); this is a paper-specific data assumption that limits generalization.
  • ad hoc to paper The damping parameter lambda=1e-2 adequately regularizes the Hessian without distorting influence rankings.
    Chosen to guarantee positive definiteness (Section 3.3); no sensitivity analysis is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Influence Functions for Preference Dataset Pruning." pith.science (2026). https://pith.science/paper/LSGZKSTA

@misc{pith2026250714344,
  author       = {Pith},
  title        = {Pith review of: Influence Functions for Preference Dataset Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LSGZKSTA}},
  note         = {Machine review of arXiv:2507.14344}
}
read the original abstract

Language models are commonly fine-tuned via reinforcement learning to alter their behavior or elicit new capabilities. Datasets used for these purposes, and particularly human preference datasets, are often noisy. The relatively small size post-training datasets, combined with parameter-efficient fine-tuning methods, enable the use of influence functions approximations to detect and prune training examples that are harmful to performance on a validation set. In this work, we adapt the TL;DR dataset for reward model training to demonstrate how conjugate-gradient approximated influence functions can be used to filter datasets. In our experiments, influence function filtering yields a small retraining accuracy uplift of 1.5% after removing 10% of training examples. We also show that gradient similarity outperforms influence functions for detecting helpful training examples. This suggests that local curvature is important for detecting harmful training examples, but less so for identifying helpful examples.

Figures

Figures reproduced from arXiv: 2507.14344 by the authors.

Figure 1
Figure 1. Test accuracy after excluding training examples. Left: removing the most helpful examples. Right: removing the most harmful ones. Vertical bars show 95 % Wald Interval [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Left: Rank correlation between Influence and Gradient Similarity Rankings (lower ranks [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 2 canonical work pages

  1. [2]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022 b

    arXiv 2022

  2. [3]

    Gradient similarity: An explainable approach to detect adversarial attacks against deep learning, 2018

    Jasjeet Dhaliwal and Saurabh Shintre. Gradient similarity: An explainable approach to detect adversarial attacks against deep learning, 2018. URL https://arxiv.org/abs/1806.10707

    arXiv 2018

  3. [4]

    Impact of preference noise on the alignment performance of generative language models, 2024

    Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment performance of generative language models, 2024. URL https://arxiv.org/abs/2404.09824

    arXiv 2024

  4. [5]

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023. URL https://arxiv.org/abs/...

    arXiv 2023

  5. [6]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022

    2022

  6. [7]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp.\ 1885--1894. PMLR, 2017

    2017

  7. [8]

    Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models

    Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models. arXiv preprint arXiv:2310.00902, 2023

    arXiv 2023

  8. [9]

    Deep learning via hessian-free optimization

    James Martens et al. Deep learning via hessian-free optimization. In Icml, volume 27, pp.\ 735--742, 2010

    work page 2010

Show all 17 references
  1. [10]

    Filtered direct preference optimization, 2024

    Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, and Kaito Ariu. Filtered direct preference optimization, 2024. URL https://arxiv.org/abs/2404.13846

    2024 arXiv

  2. [11]

    Fast exact multiplication by the hessian

    Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 6 0 (1): 0 147--160, 1994

    1994

  3. [12]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurIPS, 2020

    2020

  4. [13]

    Dataset pruning: Reducing training data by examining generalization influence, 2023

    Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence, 2023. URL https://arxiv.org/abs/2205.09329

    2023 arXiv

  5. [14]

    Star: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 0 15476--15488, 2022

    2022

  6. [15]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  7. [16]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  8. [17]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  9. [18]

    """"""""

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.