Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Spectral Insights into Data-Oblivious Critical Layers in Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A model's representation shifts reveal exactly which layers fine-tuning will rewrite, before any fine-tuning data is seen.

desk verdict Useful empirical result about instruction-tuned models, but the abstract overclaims; the correlation vanishes for a vanilla base checkpoint. read the letter →

arxiv 2506.00382 v2 pith:XNINH7RQ submitted 2025-05-31 cs.LG cs.CL

Xuyuan Liu , Lei Hsiung , Yaoqing Yang , Yujun Yan This is my paper
classification cs.LGcs.CL
keywords criticallayersrepresentationdynamicsCKACenteredKernelAlignmentfine-tuningspectralanalysisprincipalcomponentsbackdoordefense
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a large language model's most re-writable layers during supervised fine-tuning can be read off from the model itself, before any fine-tuning data is involved. The authors measure how much each layer's representations change relative to its neighbours using Centered Kernel Alignment (CKA), and they find that layers with the largest pre-fine-tuning shifts are the same layers whose substitution most damages a fine-tuned model's loss—a near-perfect negative rank correlation that repeats across five tasks for a given model. They argue this makes the critical layers data-oblivious and intrinsic to the model. The paper then shows the shifts are carried by the top three principal components, which appear to encode the transition from reasoning rationales to final conclusions, and applies the finding to two concrete settings: tuning only the critical layers accelerates domain adaptation, and freezing them cuts backdoor attack success rates by up to 40 percent. If the claim is right, fine-tuning behaviour can be predicted from a model's current internal state alone.

What carries the argument

The load-bearing object is the average CKA similarity $\delta_\ell$ of a layer $\ell$ with its $2k$ neighbours (Eq.~3), which turns raw hidden representations into a per-layer curve whose dips mark change-point layers. On the fine-tuning side, the layer-substitution loss $L(D_{\text{test}}, \tilde{\theta}/L^\ell_{\text{local}})$ measures how much the fine-tuned model depends on each local group of layers by replacing them with their pre-fine-tuned counterparts. The bridge between the two is the Spearman rank correlation, which is what establishes the claimed data-oblivious predictability. For the spectral analysis, the machinery is SVD of the centered representation matrix followed by CCA alignment of the top principal features across neighbouring layers, used to attribute the CKA dips to the second and third principal components, and a component-removal intervention that subtracts the top-K reconstruction from a layer's representation before the next layer.

What would settle it

Take an instruction-tuned model not in the paper's list (or the same models fine-tuned with a different alignment recipe), compute the CKA change-point layers on held-out prompts, then fine-tune on a fresh task and measure the Spearman correlation between the pre-fine-tuning $\delta_\ell$ and the layer-substitution loss. If the correlation is not close to -1 for that model, the claim that critical layers are intrinsically predictable from representation shifts fails. A second, sharper test: since the paper calls the layers data-oblivious, computing the CKA curve on random token sequences rather than task data should give the same change-point layers; any divergence would show the 'oblivious' property depends on the input distribution.

Watch

Extended reading notes

Core claim

The central discovery is a link between representation dynamics in pre-fine-tuned models and the layers that change most during supervised fine-tuning. In every instruction-tuned or chat model the paper tests (LLaMA-2-7B/13B-Chat, LLaMA-3.1-8B/3.2-3B-Instruct, Phi-3), the Spearman rank correlation between the average CKA similarity of a layer to its neighbours and the loss increase caused by substituting that layer with its pre-fine-tuned counterpart is close to -1 across five datasets, meaning the bigger a layer's representation shift before fine-tuning, the more fine-tuning rewrites it. The pattern is absent in the vanilla LLaMA-2-7B-Base model, indicating the effect belongs to the chat/instruct training trajectory rather than the bare architecture. Spectrally, the CKA dips at change-point layers are reproduced by the behaviour of the top three principal components; removing them at the change-point layer makes the model spell out rationales for each option, while removing only the top component changes formatting, which is read as evidence that these components condense rationales into conclusions. Two applications follow: fine-tuning only the critical layers reaches lower loss faster than fine-tuning non-critical layers, and freezing them during a poisoned fine-tuning run reduces attack success rate from 35.0 percent to below 10 percent on LLaMA-2-7B-Chat.

Load-bearing premise

The load-bearing assumption is that instruction-tuned checkpoints such as LLaMA-2/3 chat and instruct models are the relevant class of pre-fine-tuned models; since the correlation nearly vanishes for the vanilla base model, the claimed link may be a property of the chat/instruct training trajectory rather than of LLM architecture or of arbitrary pre-fine-tuned checkpoints.

Editorial extensions

If this is right

  • Fine-tuning behaviour can be predicted from a model's current state alone, without access to the fine-tuning data, so layer-wise training budgets can be set before any training run.
  • Fine-tuning only the layers with the largest representation shifts achieves lower test loss faster than tuning the least-shifted layers, approaching full fine-tuning performance.
  • Freezing the critical layers during a backdoor-poisoned fine-tuning run reduces attack success rate by up to roughly 40 percentage points, with the effect holding across two different trigger phrases.
  • Because the critical layers are consistent across tasks for a given model, one CKA pass on any reference data set suffices to locate them for a new task.
  • The top three principal components at change-point layers carry the semantic move from rationales to conclusions, so editing those components can alter reasoning behaviour while leaving other layers' contributions intact.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same CKA-based change-point analysis could be run on the pre-training trajectory itself to predict which layers a continued-pretraining run will rewrite, which would extend the claim one step earlier in the training pipeline.
  • The near-zero correlation for the base model suggests the effect may be created by the instruction-tuning process; testing chat checkpoints that were aligned with very different recipes would show whether the phenomenon is tied to a particular alignment family.
  • The paper's data-obliviousness evidence rests on five similar instruction-following and QA data sets; a stronger test would be whether the change-point curve survives random token inputs or out-of-domain text, which would separate intrinsic geometry from task-induced statistics.
  • If the second and third principal components really condense rationales into conclusions, a testable extension is to steer a model's reasoning length or confidence by scaling those components at the change-point layer, without any fine-tuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a data-oblivious method for identifying "critical layers" in large language models before supervised fine-tuning (SFT). It computes a per-layer average CKA similarity delta_l on pre-fine-tuned models, identifies layers with sharp representation shifts, and shows that these layers tend to undergo the largest changes during SFT, as measured by a layer-substitution loss. The correlation is documented in Table 2 for five chat/instruct models and several datasets, with the vanilla LLaMA-2-7B-Base included as a near-zero baseline. The paper also performs a spectral analysis attributing the CKA shifts to the top principal components, presents qualitative evidence that these components encode formatting and rationale-to-conclusion semantics, and demonstrates two applications: efficient domain adaptation by fine-tuning only critical layers, and a backdoor defense by freezing them.

Significance. If the core correlation holds for a well-defined model class, the paper makes a useful empirical contribution: it offers a way to predict which layers will be most modified during SFT using only the model's pre-fine-tuning representations, without access to fine-tuning data. The measurements of delta_l and the layer-substitution loss are made independently, which avoids the most obvious circularity, and the paper is transparent in reporting the vanishing correlation for the LLaMA-2-7B-Base baseline. The two applications, especially the backdoor-defense result with reduced attack success rates, give practical value to the finding. However, the central claim is currently scoped more broadly than the evidence supports: the positive results all come from instruction-tuned/chat models, while the one vanilla pretrained model tested shows no effect. The spectral and semantic conclusions also rely on a small number of qualitative examples and on a principal-component-removal operation that appears to be implemented differently from how it is described.

major comments (4)
  1. [Section 3.3, Table 2] The paper's unqualified claim that 'layers exhibiting greater shifts in representation space prior to fine-tuning tend to undergo more significant modifications' holds for 'pre-fine-tuned LLMs' is contradicted by its own baseline row: for LLaMA-2-7B-Base the Spearman correlations range from -0.311 to 0.147, i.e., essentially zero. Because this model shares the same architecture as LLaMA-2-7B-Chat and is an intermediate checkpoint on the path to it, the phenomenon appears to be a property of the chat/instruct training trajectory rather than an intrinsic property of pretrained LLMs generally. The title, abstract, Section 1, and Section 7 use the general 'pre-fine-tuned'/'intrinsic' language; please narrow these claims to instruction-tuned or initially fine-tuned models, or provide a criterion that determines the class of models for which the prediction is valid, since the usefulness of the predictor depends on knowing that boundary in advance.
  2. [Section 4.2, Eqs. (6)-(7)] The principal-component removal in the case study is not the operation described. Eq. (6) reconstructs the top-K part of the centered representation matrix from the SVD in Eq. (4), but Eq. (7) subtracts this from the uncentered representation x_i. The result x_{i,cleanK} therefore retains the layer mean and is not the original representation with the top-K principal components removed. The semantic conclusions about Top1 and Top3 components depend on this operation; please either include the mean in the removed component (x_i - (mu + Delta x_{i,TopK})) or work entirely in centered coordinates, and state explicitly which variant was used.
  3. [Section 5.1, Fig. 5] The efficient domain-adaptation claim is supported only by training/test loss curves for the first 50 steps; there is no final evaluation of downstream task performance (accuracy or similar) after adaptation. A reader cannot tell whether the lower loss for critical-layer fine-tuning translates into better or comparable task performance. Please report end-of-training metrics on the target datasets, or explicitly limit the claim to loss reduction.
  4. [Section 3.3, Table 2] The text states that 'consistently strong negative correlation (close to -1)' is observed for the pre-fine-tuned models, but Table 2 contains several values around -0.5, for example Phi-3-mini on Dolly (-0.533), Phi-3-mini on BoolQ (-0.538), and LLaMA-3.2-3B-Instruct on OpenBookQA (-0.509). Please qualify the claim to 'consistently negative, with strength varying by model and dataset,' or report the distribution of correlations rather than highlighting only the strongest values.
minor comments (5)
  1. [Section 4.2] The model is referred to as 'LLaMA-7B-Chat' in the text, while elsewhere it is 'LLaMA-2-7B-Chat'; please standardize the name.
  2. [Figure 2] The right panel of Figure 2 shows a colorbar ranging from -1 to 1, but the displayed pairwise Spearman correlations are all positive; please check the color scale and caption.
  3. [Section 3.1, Eq. (2)] The notation for the loss after layer substitution is written in several slightly different forms (L(D_test, theta-tilde/L^l_local) vs. L(D_test; theta-tilde/L^l_local)); please define the notation once and use it consistently.
  4. [Table 3] The caption says 'second and third eigenvectors contribute' while the table reports correlations for the Top3 principal components; please clarify whether Top1 is included in the reported values or whether the claim refers only to components two and three.
  5. [Throughout] The term 'pre-fine-tuned' is nonstandard and can be confusing; consider defining it explicitly at first use as 'the model state before the SFT stage under study,' and perhaps use 'pre-SFT' for brevity.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the paper's central correlation relates independently measured pre-fine-tuned representation shifts to post-fine-tuning substitution loss; the base-model null result is a scope caveat, not a circular step.

full rationale

The load-bearing claim (Section 3.3, Table 2) is an empirical Spearman correlation between delta_l, computed from CKA of the pre-fine-tuned model alone, and L(D_test, theta_tilde/L_local), computed by substituting pre-fine-tuned layers into the fine-tuned model. Neither quantity is defined in terms of the other, and no parameter is fitted to maximize the correlation. The data-obliviousness claim is supported by cross-dataset rank correlations (Table 1), not by construction. The spectral analysis (Section 4) reports correlations between CCA of principal components and CKA, again empirical. Applications select layers from a held-out dataset (Dolly) before attack evaluation, so the defense result is not fitted to the test data. The authors' self-citations (e.g., Liu et al. 2024c, Hsiung et al. 2025) are contextual and not load-bearing. The strongest caveat is the LLaMA-2-7B-Base row in Table 2, where the correlation vanishes; this weakens the unqualified 'pre-fine-tuned LLM' generalization and is an explicit scope/correctness risk, but it is not a circularity because the correlation was still computed honestly from independent measurements. Section 8 also acknowledges a missing theoretical foundation. No circular step can be exhibited, so score 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central correlation does not depend on fitted numerical parameters, but the paper relies on several domain assumptions: CKA on last-token representations is a meaningful measure of layer dynamics; layer substitution loss identifies true critical layers; and instruction-tuned checkpoints stand in for 'pre-fine-tuned' LLMs. The latter is the most fragile because the base model shows no correlation. The applications introduce hand-chosen budgets (five layers, top-K components), and the spectral interpretation selects Top3 after seeing the data, with Alpaca as a post hoc exception. No invented entities are introduced.

free parameters (3)
  • Neighborhood size k in CKA and layer substitution = k=2 (default; robustness checked for k=1..3)
    Defines L_local in Eq. (3) and the substitution groups in Section 3.1; changes the smoothness of delta_l and the granularity of critical layer identification.
  • Number of critical layers selected in applications = 5
    Sections 5.1 and 5.2 freeze or tune the top five critical/non-critical layers; the reported loss and ASR differences depend on this budget.
  • TopK principal components for spectral interpretation = Top3, with Top1 and Top10 as reference
    Section 4.1 selects Top3 after comparing CCA curves; for Alpaca the paper switches to Top4 (correlation 0.788) when Top3 gives 0.413, indicating post hoc selection.
assumptions (5)
  • domain assumption CKA similarity between neighboring layers' last-token representations measures meaningful representation dynamics.
    Used throughout Section 3.2 to define delta_l; follows Kornblith et al. but treats CKA as the ground truth for representation shift.
  • domain assumption Last-token hidden states are sufficient to track layer-wise information flow.
    Invoked in Section 2 with reference to Zou et al. (2023a) and Raffel et al. (2020); ignores attention over earlier tokens.
  • domain assumption Loss increase after substituting fine-tuned layers with pre-fine-tuned counterparts identifies critical layers.
    Core of Section 3.1, inspired by Zhuang et al. (2024); assumes the substitution operation is a valid counterfactual probe.
  • domain assumption Instruction-tuned checkpoints are representative of 'pre-fine-tuned' LLMs.
    Underlies the abstract and conclusion; contradicted for LLaMA-2-7B-Base in Table 2, so the scope is narrower than stated.
  • domain assumption SVD principal components of a batch of test representations correspond to interpretable semantic axes.
    Used in Section 4.2 to remove TopK components and attribute formatting and summarization functions to them; only qualitative evidence is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spectral Insights into Data-Oblivious Critical Layers in Large Language Models." pith.science (2026). https://pith.science/paper/XNINH7RQ

@misc{pith2026250600382,
  author       = {Pith},
  title        = {Pith review of: Spectral Insights into Data-Oblivious Critical Layers in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNINH7RQ}},
  note         = {Machine review of arXiv:2506.00382}
}
read the original abstract

Understanding how feature representations evolve across layers in large language models (LLMs) is key to improving their interpretability and robustness. While recent studies have identified critical layers linked to specific functions or behaviors, these efforts typically rely on data-dependent analyses of fine-tuned models, limiting their use to post-hoc settings. In contrast, we introduce a data-oblivious approach to identify intrinsic critical layers in pre-fine-tuned LLMs by analyzing representation dynamics via Centered Kernel Alignment(CKA). We show that layers with significant shifts in representation space are also those most affected during fine-tuning--a pattern that holds consistently across tasks for a given model. Our spectral analysis further reveals that these shifts are driven by changes in the top principal components, which encode semantic transitions from rationales to conclusions. We further apply these findings to two practical scenarios: efficient domain adaptation, where fine-tuning critical layers leads to greater loss reduction compared to non-critical layers; and backdoor defense, where freezing them reduces attack success rates by up to 40%.

Figures

Figures reproduced from arXiv: 2506.00382 by the authors.

Figure 1
Figure 1. Change-point layers, identified through CKA analysis, correspond to critical layers that are most [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Left: Average CKA similarity (δ ℓ ) across the layers of the LLaMA2-7B-Chat model using different test sets. Right: Pairwise Spearman’s rank correlation of δ ℓ across different test sets. Different datasets exhibit a consistent pattern in δ ℓ , suggesting its data-invariant property [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Average CCA correlation of the TopK Principal Components across layers in the LLaMA2-7B-Chat [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Case study illustrating the effects of removing different principal components at various layers. At [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Test loss curves for fine-tuning LLaMA-2-7B-Chat on the Dolly and OpenBookQA datasets by training [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualization of the loss change L  Dtest , θ˜/Lℓ local  across various models on different datasets. The same model applied to different datasets exhibits similar patterns in critical layers during the SFT stage, while different models applied to the same dataset di…
Figure 7
Figure 7. Figure 7: Visualization of the CKA similarity across different models and datasets. On the left, the heatmap displays [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 16 canonical work pages

  1. [1]

    Mitchell

    Amos Azaria and Tom M. Mitchell. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.68 The internal state of an LLM knows when it's lying . In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , pages 967--976. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.68 2023

  2. [2]

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. https://doi.org/10.48550/ARXIV.2303.08112 Eliciting latent predictions from transformers with the tuned lens . CoRR, abs/2303.08112

    arXiv doi:10.48550/arxiv.2303.08112 2023

  3. [3]

    Tu, and Henry Kvinge

    Davis Brown, Charles Godfrey, Nicholas Konz, Jonathan H. Tu, and Henry Kvinge. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.403 Understanding the inner-workings of language models through representation dissimilarity . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023...

    doi:10.18653/v1/2023.emnlp-main.403 2023

  4. [4]

    Qin Chen, Yuanyi Ren, Xiaojun Ma, and Yuyang Shi. 2025. Large language models for predictive analysis: How far are they? arXiv preprint arXiv:2505.17149

    arXiv 2025

  5. [5]

    Tianxiang Chen, Zhentao Tan, Tao Gong, Yue Wu, Qi Chu, Bin Liu, Jieping Ye, and Nenghai Yu. 2024. https://aclanthology.org/2024.findings-emnlp.347 Llama slayer 8b: Shallow layers hold the key to knowledge injection . In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024 , pages 5991--6002. Asso...

    work page 2024

  6. [6]

    Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. https://doi.org/10.18653/V1/2023.ACL-LONG.893 Analyzing transformers in embedding space . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 , pages 16124--16170. Association for Computation...

    doi:10.18653/v1/2023.acl-long.893 2023

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield - Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott J...

    arXiv doi:10.48550/arxiv.2209.07858 2022

Show all 46 references
  1. [9]

    GemmaTeam, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  2. [10]

    Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. https://doi.org/10.18653/V1/2022.EMNLP-MAIN.3 Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space . In Proceedings of the 2022 Conference on Empirical Methods in Natura...

    2022 doi

  3. [11]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. https://doi.org/10.18653/V1/2021.EMNLP-MAIN.446 Transformer feed-forward layers are key-value memories . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtua...

    2021 doi

  4. [12]

    Hardoon, S \' a ndor Szedm \' a k, and John Shawe - Taylor

    David R. Hardoon, S \' a ndor Szedm \' a k, and John Shawe - Taylor. 2004. https://doi.org/10.1162/0899766042321814 Canonical correlation analysis: An overview with application to learning methods . Neural Comput., 16(12):2639--2664

    2004 doi

  5. [13]

    Lei Hsiung, Tianyu Pang, Yung-Chen Tang, Linyue Song, Tsung-Yi Ho, Pin-Yu Chen, and Yaoqing Yang. 2025. https://openreview.net/forum?id=vQ0zFYJaMo Your task may vary: A systematic understanding of alignment and safety degradation when fine-tuning LLM s

    2025

  6. [14]

    Arthur Jacot, Cl \' e ment Hongler, and Franck Gabriel. 2018. https://proceedings.neurips.cc/paper/2018/hash/5a4be1fa34e62bb8a6ec6b91d2462f5a-Abstract.html Neural tangent kernel: Convergence and generalization in neural networks . In Advances in Neural Information Processing S...

    2018

  7. [15]

    Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. 2025. https://aclanthology.org/2025.coling-main.37/ Exploring concept depth: How large language models acqu...

    2025

  8. [16]

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. 2019. http://proceedings.mlr.press/v97/kornblith19a.html Similarity of neural network representations revisited . In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 Ju...

    2019

  9. [17]

    Ming Li, Yanhong Li, and Tianyi Zhou. 2024. https://doi.org/10.48550/ARXIV.2410.23743 What happened in llms layers when trained for fast vs. slow thinking: A gradient perspective . CoRR, abs/2410.23743

    2024 doi arXiv

  10. [18]

    Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. 2025. https://openreview.net/forum?id=kUH1yPMAn7 Safety layers in aligned large language models: The key to LLM security . In The Thirteenth International Conference on Learning Representations

    2025

  11. [19]

    Xuyuan Liu, Yinghao Cai, Qihui Yang, and Yujun Yan. 2024 a . http://papers.nips.cc/paper\_files/paper/2024/hash/f631e778fd3c1b871e9e3a94369335e9-Abstract-Conference.html Exploring consistency in graph representations: from graph kernels to graph neural networks . In Advances i...

    2024

  12. [20]

    Zhu Liu, Cunliang Kong, Ying Liu, and Maosong Sun. 2024 b . https://doi.org/10.18653/V1/2024.FINDINGS-ACL.866 Fantastic semantics and where to find them: Investigating which layers of generative llms reflect lexical semantics . In Findings of the Association for Computational ...

    2024 doi

  13. [21]

    Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, and Yaoqing Yang. 2024 c . Model balancing helps low-data training and fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1311--1331

    2024

  14. [22]

    Richard Diehl Martinez, Pietro Lesci, and Paula Buttery. 2024. https://aclanthology.org/2024.findings-emnlp.187 Tending towards stability: Convergence challenges in small language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida...

    2024

  15. [23]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html Locating and editing factual associations in GPT . In Advances in Neural Information Processing System...

    2022

  16. [24]

    Morcos, Maithra Raghu, and Samy Bengio

    Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018. https://proceedings.neurips.cc/paper/2018/hash/a7a3d70c6d17a73140918996d03c014f-Abstract.html Insights on representational similarity in neural networks with canonical correlation . In Advances in Neural Information Processi...

    2018

  17. [25]

    Thao Nguyen, Maithra Raghu, and Simon Kornblith. 2021. https://openreview.net/forum?id=KJNcAkY8tY4 Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth . In 9th International Conference on Learning Representat...

    2021

  18. [26]

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024. https://openreview.net/forum?id=L8ifDX5XNq LISA : Layerwise importance sampling for memory-efficient large language model fine-tuning . In The Thirty-eighth Annual Conference on Neural Inf...

    2024

  19. [27]

    Jason Phang, Haokun Liu, and Samuel R. Bowman. 2021. https://doi.org/10.18653/v1/2021.blackboxnlp-1.42 Fine-tuned transformers show clusters of similar representations across layers . In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networ...

    2021 doi

  20. [28]

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin - Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. https://openreview.net/forum?id=hTEGyKf0dZ Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference o...

    2024

  21. [29]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. https://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . J. Mach. Learn. Res.,...

    2020

  22. [30]

    Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl - Dickstein. 2017. https://proceedings.neurips.cc/paper/2017/hash/dc6a7e655d7e5840e66733e9ee67cc69-Abstract.html SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability ...

    2017

  23. [31]

    Sutherland

    Yi Ren and Danica J. Sutherland. 2025. https://openreview.net/forum?id=tPNHOoZFl9 Learning dynamics of LLM finetuning . In The Thirteenth International Conference on Learning Representations

    2025

  24. [32]

    Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. 2024. https://doi.org/10.18653/v1/2024.acl-long.111 V alue B ench: Towards comprehensively evaluating value orientations and understanding of large language models . In Proceedings of the 62nd Annual Meeting of th...

    2024 doi

  25. [33]

    Oscar Skean, Md Rifat Arefin, Yann LeCun, and Ravid Shwartz - Ziv. 2024. https://doi.org/10.48550/ARXIV.2412.09563 Does representation matter? exploring intermediate layers in large language models . CoRR, abs/2412.09563

    2024 doi arXiv

  26. [34]

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. https://openreview.net/forum?id=F7aAhfitX6 Massive activations in large language models . In First Conference on Language Modeling

    2024

  27. [35]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html Attention is all you need . In Advances in Neural I...

    2017

  28. [36]

    Elena Voita, Rico Sennrich, and Ivan Titov. 2019. https://doi.org/10.18653/V1/D19-1448 The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives . In Proceedings of the 2019 Conference on Empirical Methods ...

    2019 doi

  29. [37]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. https://openreview.net/forum?id=gEZrGCozdqR Finetuned language models are zero-shot learners . In The Tenth International Conference on Learning Repr...

    2022

  30. [38]

    Haoran Ye, Yuhang Xie, Yuanyi Ren, Hanjun Fang, Xin Zhang, and Guojie Song. 2025. Measuring human and ai values based on generative psychometrics with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26400--26408

    2025

  31. [39]

    Yi Zeng, Weiyu Sun, Tran Ngoc Huynh, Dawn Song, Bo Li, and Ruoxi Jia. 2024. https://aclanthology.org/2024.emnlp-main.732 BEEAR: embedding-based adversarial removal of safety backdoors in instruction-tuned language models . In Proceedings of the 2024 Conference on Empirical Met...

    2024

  32. [40]

    Wei Zhao, Zhe Li, Yige Li, Ye Zhang, and Jun Sun. 2024. https://aclanthology.org/2024.findings-emnlp.293 Defending large language models against jailbreak attacks via layer-specific editing . In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Flor...

    2024

  33. [41]

    Martin, Michael W

    Yefan Zhou, Tianyu Pang, Keqin Liu, Charles H. Martin, Michael W. Mahoney, and Yaoqing Yang. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/c8a4dd7d9e13583d714ce8580da7bbc7-Abstract-Conference.html Temperature balancing, layer-wise weight analysis, and neural network...

    2023

  34. [42]

    Haomin Zhuang, Mingxian Yu, Hao Wang, Yang Hua, Jian Li, and Xu Yuan. 2024. https://openreview.net/forum?id=AJBGSVSTT2 Backdoor federated learning by poisoning backdoor-critical layers . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Au...

    2024

  35. [43]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann - Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zi...

    2023 doi arXiv

  36. [44]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023 b . https://doi.org/10.48550/ARXIV.2307.15043 Universal and transferable adversarial attacks on aligned language models . CoRR, abs/2307.15043

    2023 doi arXiv

  37. [45]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.