Pith. sign in

REVIEW 4 major objections 5 minor 82 references

TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims every fine-tuned LLM decomposes exactly into pre-training and fine-tuning parts, and the ratio TuCo measures how much fine-tuning shaped each response.

desk verdict TuCo is a genuinely useful per-prompt metric with strong cross-model empirical results, but the paper's central claim that it measures fine-tuning's causal contribution is unvalidated and partly confounded with distribution shift. read the letter →

arxiv 2506.23423 v1 pith:7YBPFCQ4 submitted 2025-06-29 cs.CL cs.AI

Felipe Nuti , Tim Franzmeyer , João Henriques This is my paper · ORCID
classification cs.CLcs.AI
keywords TuningContributionfine-tuningattributionresidualstreamdecompositioninterpretabilityjailbreakattacksmany-shotjailbreakingconjugatepromptingLLMsafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that any fine-tuned transformer LLM can be decomposed exactly, layer by layer, into a pre-training component (the output of the corresponding pre-trained layer) and a fine-tuning component (the difference between the fine-tuned and pre-trained layer outputs). Building on that identity, it defines TuCo as the ratio of the accumulated last-token magnitudes of the fine-tuning component to the pre-training component, and interprets TuCo as the fraction of the response attributable to fine-tuning. Experimentally, the paper shows that scaling the fine-tuning component during the forward pass changes model behavior and benchmark accuracy, and that three jailbreak techniques—adversarial suffix attacks, translation to low-resource languages, and many-shot prompting—all reduce TuCo, more so when the attack succeeds. If correct, TuCo provides a per-prompt, inference-time measurement of fine-tuning's influence, with direct relevance to interpretability and to understanding how jailbreaks bypass safety training.

What carries the argument

The load-bearing object is the canonical decomposition of a fine-tuned transformer into two generalized components: PTC, the output of the pre-trained layer on the same residual-stream input, and FTC, the difference between the fine-tuned and pre-trained layer outputs. Because every residual layer adds its output to the stream, the accumulated sums $\mathrm{PTC}_l$ and $\mathrm{FTC}_l$ make the final hidden state an exact sum of a pre-training and a fine-tuning contribution. A discrete Grönwall inequality converts the ratio of these accumulated magnitudes into a bound on how far the fine-tuned final state can drift from the pre-trained one, and TuCo is the last-token form of that ratio, chosen so that pre-training and fine-tuning contributions sum to one.

What would settle it

Take a harmless prompt and its translation into a low-resource language such as Swahili or Malayalam; if TuCo drops just as much for the benign translation as for the harmful one while the model's helpfulness and refusal behavior are unchanged, then TuCo is tracking distribution shift rather than the causal contribution of fine-tuning.

Watch

Extended reading notes

Core claim

The central discovery is the exact canonical decomposition $f^{\mathrm{FT}}_\Theta(x,l) = \mathrm{PTC}(x,l) + \mathrm{FTC}(x,l)$ with $\mathrm{PTC}(x,l) = f^{\mathrm{PT}}_\phi(x,l)$ and $\mathrm{FTC}(x,l) = f^{\mathrm{FT}}_\Theta(x,l) - f^{\mathrm{PT}}_\phi(x,l)$. Because transformer layers write additively to the residual stream, the final hidden state satisfies $x_L = x_0 + \mathrm{PTC}_L + \mathrm{FTC}_L$, and a discrete Grönwall bound shows that the ratio $\beta_l = \|\mathrm{FTC}_l\|_1/(\|\mathrm{PTC}_l\|_1 + \|\mathrm{FTC}_l\|_1)$ controls the distance between the pre-trained and fine-tuned final hidden states. TuCo is the last-token version of this ratio, $\mathrm{TuCo}(x) = \|\mathrm{proj}_n(\mathrm{FTC}_L)\|_1/(\|\mathrm{proj}_n(\mathrm{PTC}_L)\|_1 + \|\mathrm{proj}_n(\mathrm{FTC}_L)\|_1)$. The paper's empirical claim is that this scalar is behaviorally meaningful: multiplying the fine-tuning component by $\alpha \in [0.75, 1.25]$ shifts MMLU accuracy by up to about 5% and agreement with Christian beliefs by up to 24% on Llama 2 13B, and three jailbreak families make TuCo drop, with AUC 0.87 separating successful from unsuccessful attacks on Llama 2 13B.

Load-bearing premise

The load-bearing assumption is that the magnitude of the fine-tuning component—the difference between fine-tuned and pre-trained layer outputs—reflects how much fine-tuning actually causes the response, rather than merely how unusual the input is for the fine-tuned model.

Editorial extensions

If this is right

  • Scaling the fine-tuning component during the forward pass is a control knob: on MMLU, 71% of tasks improve out-of-sample with average gains of 1.03–2.69%, and agreement with Christian beliefs rises by 24% for Llama 2 13B when the component is scaled by $\alpha = 1.25$.
  • All three jailbreak families reduce TuCo, and increasing many-shot attack intensity lowers TuCo monotonically, which the paper interprets as the attack implicitly applying FTC-scaling with $\alpha < 1$.
  • Successful jailbreaks have lower TuCo than unsuccessful ones (AUC 0.87 for Llama 2 13B and above 0.8 for most other tested models), so TuCo can be used as an analysis signal for where safety-tuning has lost its grip.
  • TuCo separates pre-training-like web text from chat-like fine-tuning inputs with AUC above 0.8 for eight of nine models, supporting the claim that it tracks the fine-tuning regime rather than prompt length or perplexity.
  • TuCo is computable at inference time from two forward passes, so it can be applied to open billion-parameter models without training probes or gradient access.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If TuCo is read as an anomaly score rather than a causal measure, the conjugate-prompting result predicts that any out-of-distribution input, even a harmless one, should lower TuCo; this is directly testable on benign prompts translated to Swahili or Malayalam.
  • The FTC-scaling result opens the possibility of an inference-time 'fine-tuning dial' for open models, but the paper does not propose it as a defense, and it would not transfer to closed models where the pre-trained base is unavailable.
  • A natural extension is to compute layer-level or token-level TuCo maps to localize where fine-tuning's contribution enters the forward pass; the paper only defines the scalar per-prompt value.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TuCo, a per-prompt metric intended to quantify how much fine-tuning contributes to an LLM's individual responses. The method assumes access to both a pre-trained model and its fine-tuned counterpart. The core algebraic observation, formalized in Proposition D.3, is that at every residual layer the fine-tuned layer output can be written exactly as the pre-trained layer output evaluated on the fine-tuned trajectory (PTC) plus the difference between the fine-tuned and pre-trained layer outputs (FTC). The paper proves a discrete Grönwall bound (Proposition 4.2) showing that the maximum over layers of a certain ratio of cumulative FTC and PTC norms controls the distance between final hidden states of the two models. TuCo is then defined in Definition 4.3 as the norm of the cumulative last-token FTC divided by the sum of the norms of the cumulative last-token PTC and FTC. Empirically, the paper shows that scaling FTC by a factor α can modulate model behavior on Model-Written Evaluations and MMLU; that TuCo is lower for OpenWebText than for HH-RLHF chats; that three jailbreak attacks (GCG, conjugate prompting, many-shot) are reported to reduce TuCo; and that TuCo is lower on successful jailbreaks for most tested models. The paper interprets these results as quantitative evidence that jailbreaks attenuate the effect of fine-tuning during the forward pass.

Significance. If the central interpretive claim were validated, TuCo would be a valuable, inexpensive interpretability tool: it requires only two forward passes and no training, and the layer-wise decomposition is exact and model-agnostic. The alpha-scaling experiments are a genuinely interesting intervention, showing that the FTC direction carries behaviorally relevant signal for several behaviors and tasks. The paper is also commendable for releasing code and for being explicit about the requirements and limitations of the method. However, the significance is currently limited by an unvalidated leap from 'FTC norm is large' to 'fine-tuning contributes to this response.' The paper's own example in Appendix B shows that TuCo can be nonzero when the fine-tuned and pre-trained models produce identical final hidden states (and hence identical sampling distributions), which directly contradicts the name and interpretation of the metric as measuring contribution to a response. Several empirical claims, notably the claim that all three jailbreak attacks reduce TuCo, are not supported by the reported per-model AUCs.

major comments (4)
  1. [§4.4–4.5, Proposition 4.2 and Definition 4.3] Proposition 4.2 bounds the final-state discrepancy in terms of β = max_{0≤l<L} ∥FTC_l∥ / (∥PTC_l∥ + ∥FTC_l∥), but TuCo is defined using β_L (the final-layer cumulative ratio) and only the last token. The theorem does not imply a bound or guarantee for TuCo. The discussion in §4.5 and Appendix A justifies these changes by symmetry and interpretability, but no matching theorem or even a formal heuristic is supplied. As written, the theoretical analysis does not support the claim that a low TuCo means fine-tuning has little effect on the response; it supports a statement about a different quantity. Please either prove a similar bound for TuCo (or a related quantity), or clearly state in the main text that TuCo is a heuristic metric motivated by, but not implied by, Proposition 4.2.
  2. [§4.1 and Appendix B, Example B.1] Example B.1 is a two-layer construction where the fine-tuned and pre-trained models have identical final hidden states, yet TuCo = 1/4. Since the final hidden state determines the next-token distribution, the response is identical in both models, so fine-tuning makes no causal contribution to the response on this input. The paper embraces this as a feature (capturing internal activity), but it undermines the metric's name and its central interpretation as measuring the contribution of fine-tuning to a response. The alpha-scaling experiments in §5.1 show that the FTC direction contains task-relevant signal in aggregate, but they do not establish that a prompt's TuCo value predicts how much fine-tuning actually changes that prompt's output. Please add a per-prompt validation: for example, correlate TuCo with the change in output distribution when FTC is scaled or removed, or with the actual distance between fine-tuned and pre-trained output distributions across a diverse prompt set.
  3. [§5.3, Table 1] The sentence 'We find that all three attacks significantly decrease TuCo when applied to harmful prompts' is not supported by the CP columns of Table 1. For Gemma 7B the Japanese-vs-low-resource AUC is 0.12, for Zephyr Gemma the Japanese AUC is 0.23 and Hungarian is 0.19, and for Llama 3 8B the Hungarian AUC is 0.40. These values indicate the opposite direction for a substantial fraction of model/language pairs. The clean web-share ordering is demonstrated only for Llama 2 13B. Please report per-language-pair statistics with confidence intervals, and either soften the claim to the models/languages where the effect holds, or explain why the reversed AUCs are consistent with the overall conclusion.
  4. [§5.2–5.4] The empirical comparisons are confounded by distribution shift and prompt length. OpenWebText and HH-RLHF differ in many ways beyond 'pre-training-like' versus 'chat-like'; many-shot jailbreaks increase prompt length; conjugate prompting changes language and token distribution; GCG adds an adversarial suffix. Since TuCo is lower on OpenWebText, the observed decreases under jailbreaks could be an anomaly-score effect — prompts far from the fine-tuning distribution tend to have smaller layer-wise differences — rather than attenuation of fine-tuning's causal role. The statement in §5.4 that TuCo is 'not merely reflecting spurious aspects of the prompt (e.g. length or perplexity)' is not backed by any control. Please include explicit controls (e.g., length-matched prompts, perplexity-matched non-attack prompts, or regression analyses partialling out these variables) so that the causal interpretation is not confounded.
minor comments (5)
  1. [§5.2, Table 1] The text says the AUC is above 0.80 for all but two models, but the table shows only Zephyr Gemma (0.63) below 0.80; please correct the count.
  2. [§5.1, Appendix F.1] The statistical significance claims for MWE and MMLU are reported without standard errors, p-values, or multiple-testing correction; please provide these details or clarify the test used.
  3. [Algorithm 1] Algorithm 1's comment 'Compute PTC for layer l' could be misinterpreted as running the pre-trained model's own forward pass; the fact that PTC is evaluated on the fine-tuned trajectory is explained in the text but should be emphasized in the algorithm caption as well.
  4. [Appendix D.4] There are several typographical issues in the proof of Corollary D.5 (e.g., malformed summation indices) and in Appendix D.5 ('traingle inequality', 'Propostion'); please proofread the appendix carefully.
  5. [§5.3] For many-shot jailbreaking, the paper reports a downward trend in TuCo with the number of shots, but the prompt length also increases with the number of shots; a length-controlled comparison would strengthen the interpretation.

Circularity Check

1 steps flagged · score 4.0 of 10

TuCo is defined as a norm ratio of a layer-wise difference, and the paper's central claim that it 'measures the contribution of fine-tuning' is true by that definition, though independent scaling and AUC experiments give the metric empirical content.

  1. self definitional [Section 4.5, Definition 4.3; Proposition D.3; Appendix A]
    "FTC(x, l) := f_FT_theta(x, l) - f_PT_phi(x, l). ... TuCo(x) := ||projn(FTCL)||_1 / (||projn(PTCL)||_1 + ||projn(FTCL)||_1). ... Since the final hidden state is given by xL = x0 + PTCL + FTCL, and TuCo = ..., we can interpret TuCo as the 'fraction' of the final hidden state that is attributable to the fine-tuning component."

    The quantity the paper claims to measure, 'the contribution of fine-tuning to an individual response,' is never given an independent operationalization. Instead, 'contribution' is defined to be the norm ratio of FTC, and FTC itself is defined as the layer-wise output difference between the fine-tuned and pre-trained models. Thus the statement that TuCo measures fine-tuning contribution is true by construction, not by empirical validation. In particular, the jailbreak conclusion that attacks 'attenuate the effect of fine-tuning' is a restatement, through this definition, of the observation that the norm of this defined difference is smaller on such prompts.

full rationale

The algebraic decomposition in Proposition D.3 is exact but tautological: since FTC is defined as f_FT - f_PT, the identity f_FT = PTC + FTC holds by construction. TuCo is then defined from these components, and the paper's core interpretive claim equates TuCo with 'tuning contribution.' This is a self-definitional step: no ground-truth notion of contribution is defined independently of TuCo, so the claim that TuCo measures contribution reduces to its definition. However, the paper does not stop at the definition. The FTC alpha-scaling experiments (Section 5.1) show that scaling the FTC direction alters model behavior and MMLU/MWE agreement, which is independent evidence that this activation-space component carries behavioral signal. The empirical AUC results for distinguishing successful from unsuccessful jailbreaks are also not forced by the definition. There are no load-bearing self-citations: the cited competing-objectives hypotheses (Kotha et al., Wei et al.) are external prior work, not the authors' own. There are no fitted parameters renamed as predictions and no imported uniqueness theorems. The circularity is therefore localized to the interpretive equation 'TuCo = contribution,' which is definitional rather than derived. Because the central metric's meaning is definitional but the empirical demonstrations provide independent content, a moderate score of 4 is appropriate.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. The free parameters are limited to the alpha scaling knob used in Section 5.1 experiments. The key axioms are the residual architecture assumption and the Lipschitz/boundedness condition on PTC, the latter being the most fragile part of the theoretical bound.

free parameters (1)
  • alpha (FTC scaling factor) = grid in [0.75, 1.25]
    In Section 5.1, alpha is chosen per task/behavior via grid search on a validation fold to maximize accuracy; the reported behavioral and MMLU improvements depend on this per-task selection, though it is cross-validated.
assumptions (4)
  • domain assumption Transformer layers are residual: x_{l+1} = x_l + f(x_l, l).
    Assumed in Section 3 and used to define PTC/FTC and the Gronwall bound.
  • domain assumption PTC is bounded and Lipschitz with respect to x.
    Needed for Proposition 4.2; justified in Appendix D.5 for RMSNorm-based layers, but not proven for all architectures.
  • standard math Discrete Gronwall inequality (Clark 1987).
    Used to prove Proposition 4.2; standard result.
  • domain assumption The pre-trained and fine-tuned models share the same architecture, layer count, and tokenizer.
    TuCo requires running corresponding layers of both models on the same intermediate hidden states; this holds only when architectures match.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs." pith.science (2026). https://pith.science/paper/7YBPFCQ4

@misc{pith2026250623423,
  author       = {Pith},
  title        = {Pith review of: TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7YBPFCQ4}},
  note         = {Machine review of arXiv:2506.23423}
}
read the original abstract

Past work has studied the effects of fine-tuning on large language models' (LLMs) overall performance on certain tasks. However, a quantitative and systematic method for analyzing its effect on individual outputs is still lacking. Here, we propose a new method for measuring the contribution that fine-tuning makes to individual LLM responses, assuming access to the original pre-trained model. Our method tracks the model's intermediate hidden states, providing a more fine-grained insight into the effects of fine-tuning than a simple comparison of final outputs from pre-trained and fine-tuned models. We introduce and theoretically analyze an exact decomposition of any fine-tuned LLM into a pre-training component and a fine-tuning component. Empirically, we find that model behavior and performance can be steered by up- or down-scaling the fine-tuning component during the forward pass. Motivated by this finding and our theoretical analysis, we define the Tuning Contribution (TuCo) as the ratio of the magnitudes of the fine-tuning component to the pre-training component. We observe that three prominent adversarial attacks on LLMs circumvent safety measures in a way that reduces TuCo, and that TuCo is consistently lower on prompts where these attacks succeed compared to those where they do not. This suggests that attenuating the effect of fine-tuning on model outputs plays a role in the success of such attacks. In summary, TuCo enables the quantitative study of how fine-tuning influences model behavior and safety, and vice versa.

Figures

Figures reproduced from arXiv: 2506.23423 by the authors.

Figure 1
Figure 1. On the left, we observe example prompts and responses by an LLM, which was first pre-trained and then fine-tuned. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Model behavior change for scaling the Fine [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Top two panels: Different attacks result in dis [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Top: comparison of OutputCo and TuCo on Llama 2 7B for a dataset of prompts consisting of several ex￾amples of model refusals, followed by a harmless question. Bottom: comparison of the norms of the fine-tuning compo￾nent (FTC norm) and [PITH_FULL_IMAGE:figures/full_f…
Figure 6
Figure 6. Figure 6: Additional examples of behavior change for scaling the Fine-Tuning Component by [PITH_FULL_IMAGE:figures/full_fig_p031_6.png]
Figure 7
Figure 7. Figure 7: Delta in cross-validated accuracy in MMLU tasks, broken down by model and subfield. [PITH_FULL_IMAGE:figures/full_fig_p032_7.png]
Figure 12
Figure 12. Figure 12: Delta in cross-validated accuracy in MWE behaviors when picking [PITH_FULL_IMAGE:figures/full_fig_p037_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 47 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Bengio, Y

    Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes, 2017. URL https://openreview.net/forum?id=ryF7rTqgl

    2017

  3. [3]

    and Kamfonas, M

    Alon, G. and Kamfonas, M. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132, 2023

    arXiv 2023

  4. [4]

    Many-shot jailbreaking

    Anil, C., Durmus, E., Panickssery, N., Sharma, M., Benton, J., Kundu, S., Batson, J., Tong, M., Mu, J., Ford, D., et al. Many-shot jailbreaking. Advances in Neural Information Processing Systems, 37: 0 129696--129742, 2024

    work page 2024

  5. [5]

    A general language assistant as a laboratory for alignment

    Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021

    arXiv 2021

  6. [6]

    and Mitchell, T

    Azaria, A. and Mitchell, T. The internal state of an llm knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 967--976, 2023

    work page 2023

  7. [7]

    L., Kiros, J

    Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. stat, 1050: 0 21, 2016

    work page 2016

  8. [8]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022 a

    arXiv 2022

Show all 82 references
  1. [9]

    Constitutional ai: Harmlessness from ai feedback

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022 b

    2022 arXiv

  2. [10]

    Probing classifiers: Promises, shortcomings, and advances

    Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48 0 (1), 2021

    2021

  3. [11]

    Bishop, C. M. Pattern recognition and machine learning. Springer google schola, 2: 0 645--678, 2006

    2006

  4. [12]

    Findings of the 2014 workshop on statistical machine translation

    Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pp.\ 12--58, 2014

    2014

  5. [13]

    Bradley, A. P. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern Recognition, 30 0 (7): 0 1145--1159, 1997. ISSN 0031-3203. doi:https://doi.org/10.1016/S0031-3203(96)00142-2. URL https://www.sciencedirect.com/science/article/pii/...

    1997 doi

  6. [14]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  7. [15]

    Discovering latent knowledge in language models without supervision

    Burns, C., Ye, H., Klein, D., and Steinhardt, J. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2022

    2022

  8. [16]

    T., Rubanova, Y., Bettencourt, J., and Duvenaud, D

    Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

    2018

  9. [17]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  10. [18]

    Clark, D. S. Short proof of a discrete gronwall inequality. Discrete applied mathematics, 16 0 (3): 0 279--281, 1987

    1987

  11. [19]

    Some Gronwall Type Inequalities and Applications

    Dragomir, S. Some Gronwall Type Inequalities and Applications. Nova Science Publishers, 2003. ISBN 9781590338278. URL https://books.google.co.uk/books?id=3KUrAAAAYAAJ

    2003

  12. [20]

    A mathematical framework for transformer circuits

    Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan...

    2021

  13. [21]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

    Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. CoRR, 2022

    2022

  14. [22]

    and Cohen, V

    Gokaslan, A. and Cohen, V. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  15. [23]

    Studying large language model generalization with influence functions

    Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023

    2023 arXiv

  16. [24]

    Simfluence: Modeling the influence of individual training examples by simulating training runs

    Guu, K., Webson, A., Pavlick, E., Dixon, L., Tenney, I., and Bolukbasi, T. Simfluence: Modeling the influence of individual training examples by simulating training runs. arXiv preprint arXiv:2303.08114, 2023

    2023 arXiv

  17. [25]

    and Lowd, D

    Hammoudeh, Z. and Lowd, D. Training data influence analysis and estimation: a survey. Machine Learning, 113 0 (5): 0 2351--2403, 2024. doi:10.1007/s10994-023-06495-7. URL https://doi.org/10.1007/s10994-023-06495-7

    2024 doi

  18. [26]

    Hampel, F. R. The influence curve and its role in robust estimation. Journal of the american statistical association, 69 0 (346): 0 383--393, 1974

    1974

  19. [27]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020

    2020

  20. [28]

    T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A

    Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations

  21. [29]

    Baseline defenses for adversarial attacks against aligned language models

    Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., Chiang, P.-y., Goldblum, M., Saha, A., Geiping, J., and Goldstein, T. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614, 2023

    2023 arXiv

  22. [30]

    S., Dick, R

    Jain, S., Kirk, R., Lubana, E. S., Dick, R. P., Tanaka, H., Rockt \"a schel, T., Grefenstette, E., and Krueger, D. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. In The Twelfth International Conference on Learning Representations, 2024

    2024

  23. [31]

    J., Hassani, H., Zhang, Y., Wong, E., and Chang, S

    Ji, J., Hou, B., Robey, A., Pappas, G. J., Hassani, H., Zhang, Y., Wong, E., and Chang, S. Defending large language models against jailbreak attacks via semantic smoothing, 2024

    2024

  24. [32]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  25. [33]

    Koh, P. W. and Liang, P. Understanding black-box predictions via influence functions. In International conference on machine learning, pp.\ 1885--1894. PMLR, 2017

    2017

  26. [34]

    M., and Raghunathan, A

    Kotha, S., Springer, J. M., and Raghunathan, A. Understanding catastrophic forgetting in language models via implicit inference. In The Twelfth International Conference on Learning Representations

  27. [35]

    J., Feizi, S., and Lakkaraju, H

    Kumar, A., Agarwal, C., Srinivas, S., Li, A. J., Feizi, S., and Lakkaraju, H. Certifying llm safety against adversarial prompting. In First Conference on Language Modeling

  28. [36]

    Inference-time intervention: Eliciting truthful answers from a language model

    Li, K., Patel, O., Vi \'e gas, F., Pfister, H., and Wattenberg, M. Inference-time intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  29. [37]

    Mitigating the alignment tax of rlhf

    Lin, Y., Tan, L., Lin, H., Zheng, Z., Pi, R., Zhang, J., Diao, S., Wang, H., Zhao, H., Yao, Y., and Zhang, T. Mitigating the alignment tax of rlhf. 2023. URL https://api.semanticscholar.org/CorpusID:261697277

    2023

  30. [38]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Liu, X., Xu, N., Chen, M., and Xiao, C. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Representations

  31. [39]

    S., Love, J., Tafti, P., et al

    Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi \`e re, M., Kale, M. S., Love, J., Tafti, P., et al. Gemma: Open models based on gemini research and technology. CoRR, 2024

    2024

  32. [40]

    Introducing meta llama 3: The most capable openly available llm to date

    Meta AI . Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/, 2024. Accessed: April 24, 2024

    2024

  33. [41]

    Nguyen, E., Seo, M., and Oh, S. J. A bayesian approach to analysing training data attribution in deep learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  34. [42]

    Noukhovitch, M., Lavoie, S., Strub, F., and Courville, A. C. Language model alignment with elastic reset. Advances in Neural Information Processing Systems, 36, 2024

    2024

  35. [43]

    Zoom in: An introduction to circuits

    Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom in: An introduction to circuits. Distill, 2020. doi:10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in

    2020 doi

  36. [44]

    In-context learning and induction heads

    Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Cla...

    2022

  37. [45]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  38. [46]

    Discovering language model behaviors with model-written evaluations

    Perez, E., Ringer, S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 133...

    2023

  39. [47]

    D., Peng, S., Szyller, S., Cornelius, C., and Chau, D

    Phute, M., Helbling, A., Hull, M. D., Peng, S., Szyller, S., Cornelius, C., and Chau, D. H. Llm self defense: By self examination, llms know they are being tricked. In The Second Tiny Papers Track at ICLR 2024

    2024

  40. [48]

    R., Haklay, T., Belinkov, Y., and Bau, D

    Prakash, N., Shaham, T. R., Haklay, T., Belinkov, Y., and Bau, D. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=8sKcAWOf2D

    2024

  41. [49]

    Estimating training data influence by tracing gradient descent

    Pruthi, G., Liu, F., Kale, S., and Sundararajan, M. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 0 19920--19930, 2020

    2020

  42. [50]

    Improving language understanding by generative pre-training

    Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training

  43. [51]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. URL https://api.semanticscholar.org/CorpusID:160025533

    2019

  44. [52]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  45. [53]

    Steering llama 2 via contrastive activation addition

    Rimsky, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., and Turner, A. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 15504--15522, 2024

    2024

  46. [54]

    Robey, A., Wong, E., Hassani, H., and Pappas, G. J. Smoothllm: Defending large language models against jailbreaking attacks, 2023

    2023

  47. [55]

    The perceptron: a probabilistic model for information storage and organization in the brain

    Rosenblatt, F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65 0 (6): 0 386, 1958

    1958

  48. [56]

    Principles of Mathematical Analysis

    Rudin, W. Principles of Mathematical Analysis. International series in pure and applied mathematics. McGraw-Hill, 1976. ISBN 9780070856134. URL https://books.google.co.uk/books?id=kwqzPAAACAAJ

    1976

  49. [57]

    E., Ablin, P., and Peyr \'e , G

    Sander, M. E., Ablin, P., and Peyr \'e , G. Do residual neural networks discretize neural ordinary differential equations? In Advances in Neural Information Processing Systems, 2022

    2022

  50. [58]

    Scaling up influence functions

    Schioppa, A., Zablotskaia, P., Vilar, D., and Sokolov, A. Scaling up influence functions. Proceedings of the AAAI Conference on Artificial Intelligence, 36 0 (8): 0 8179--8186, Jun. 2022. doi:10.1609/aaai.v36i8.20791. URL https://ojs.aaai.org/index.php/AAAI/article/view/20791

    2022 doi

  51. [59]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347

    2017 arXiv

  52. [60]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  53. [61]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  54. [62]

    and Schmid, P

    Tunstall, L. and Schmid, P. Zephyr 7b gemma. https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1, 2024

    2024

  55. [63]

    M., and Wolf, T

    Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., Sarrazin, N., Sanseviero, O., Rush, A. M., and Wolf, T. Zephyr: Direct distillation of lm alignment, 2023

    2023

  56. [64]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  57. [65]

    Ordinary differential equations, volume 182

    Walter, W. Ordinary differential equations, volume 182. Springer Science & Business Media, 2013

    2013

  58. [66]

    R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J

    Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations, 2022

    2022

  59. [67]

    Defending llms against jailbreaking attacks via backtranslation

    Wang, Y., Shi, Z., Bai, A., and Hsieh, C.-J. Defending llms against jailbreaking attacks via backtranslation. In Findings of the Association for Computational Linguistics ACL 2024, pp.\ 16031--16046, 2024

    2024

  60. [68]

    Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024

    Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024

    2024

  61. [69]

    Larger language models do in-context learning differently, 2023

    Wei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., and Ma, T. Larger language models do in-context learning differently, 2023

    2023

  62. [70]

    Principal component analysis

    Wold, S., Esbensen, K., and Geladi, P. Principal component analysis. Chemometrics and intelligent laboratory systems, 2 0 (1-3): 0 37--52, 1987

    1987

  63. [71]

    Usage statistics of content languages for websites

    World Wide Web Technology Surveys . Usage statistics of content languages for websites. https://w3techs.com/technologies/overview/content_language, 2024. Accessed: May 4, 2024

    2024

  64. [72]

    W., Li, M., Kornblith, S., Roelofs, R., Lopes, R

    Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 7959--7...

    2022

  65. [73]

    Defending chatgpt against jailbreak attack via self-reminders

    Xie, Y., Yi, J., Shao, J., Curl, J., Lyu, L., Chen, Q., Xie, X., and Wu, F. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5 0 (12): 0 1486--1496, 2023. doi:10.1038/s42256-023-00765-8. URL https://doi.org/10.1038/s42256-023-00765-8

    2023 doi

  66. [74]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019

    2019

  67. [75]

    and Sennrich, R

    Zhang, B. and Sennrich, R. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  68. [76]

    Intention analysis makes llms a good jailbreak defender

    Zhang, Y., Ding, L., Zhang, L., and Tao, D. Intention analysis makes llms a good jailbreak defender. In Proceedings of the 31st International Conference on Computational Linguistics, pp.\ 2947--2968, 2025

    2025

  69. [77]

    Zhang, Z., Zhang, Q., and Foerster, J. N. Parden, can you repeat that? defending against jailbreaks via repetition. In Forty-first International Conference on Machine Learning

  70. [78]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024

    2024

  71. [79]

    Defending jailbreak prompts via in-context adversarial game

    Zhou, Y., Han, Y., Zhuang, H., Guo, K., Liang, Z., Bao, H., and Zhang, X. Defending jailbreak prompts via in-context adversarial game. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 20084--20105, 2024

    2024

  72. [80]

    Autodan: Interpretable gradient-based adversarial attacks on large language models

    Zhu, S., Zhang, R., An, B., Wu, G., Barrow, J., Wang, Z., Huang, F., Nenkova, A., and Sun, T. Autodan: Interpretable gradient-based adversarial attacks on large language models. In First Conference on Language Modeling, 2023

    2023

  73. [81]

    Representation engineering: A top-down approach to ai transparency

    Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. Representation engineering: A top-down approach to ai transparency. CoRR, 2023 a

    2023

  74. [82]

    Z., and Fredrikson, M

    Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023 b

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.