Pith. sign in

REVIEW 4 major objections 7 minor 20 references

Probing the Embedding Space of Transformers via Minimal Token Perturbations

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Minimal token perturbations show that a BERT model's early layers stay closer to the original input than its deeper layers, supporting the use of early-layer representations as explanations.

desk verdict A small, readable empirical study that recombines known ingredients into a new perturbation protocol, but whose central claim about early layers as explanation proxies overreaches the evidence it provides. read the letter →

arxiv 2506.18011 v1 pith:GXJFG2KC submitted 2025-06-22 cs.LG cs.AIstat.ML

Eddie Conti , Alejandro Astruc , Alvaro Parafita , Axel Brando This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords minimaltokenperturbationsembeddingspaceTransformerinterpretabilityfrequencylayer-wisepropagationBERThiddenstateshiftsidentifiability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Using a fine-tuned BERT sentiment model, the paper replaces a single token in a movie review with its nearest embedding neighbor (by cosine similarity) and measures how far the sentence's hidden states move. It finds that the tokens whose swaps shift the embedding least are overwhelmingly common words, punctuation, and numbers, while rare tokens produce larger shifts. Tracking the same minimal perturbations through the twelve transformer layers, the distance between perturbed and original hidden states grows with depth, except in the final layer, which the authors attribute to task-specific fine-tuning. The paper takes these trends as evidence that input information becomes increasingly intermixed in deeper layers and concludes that early-layer representations are more faithful to the input and therefore safer proxies for model explanations.

What carries the argument

The central mechanism is the minimal token perturbation: for each token in an input sentence, replace it with the vocabulary token whose embedding has the highest cosine similarity, then measure the shift $\|E(x)-E(x')\|$ produced in the embedding space under the $\ell^1$, $\ell^2$, and $\ell^\infty$ norms. The same replacement is passed through the encoder to compute hidden-state distances at each layer. This construction is what allows the paper to connect token frequency, embedding sensitivity, and layer depth in one interpretability probe.

What would settle it

Take the same 500 sequences and, for the top-k tokens flagged by each layer's closest-token retrieval, erase or replace those tokens, then measure the change in the model's sentiment prediction. If attributions from deeper layers track the prediction change at least as well as early-layer ones, the claim that first layers are better explanation proxies would be contradicted; alternatively, recomputing the layer-wise perturbation curve with per-layer normalized distances would show whether the increasing trend survives after removing scale effects.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that minimal token perturbations expose a layer-wise fidelity gradient: BERT's embedding and early hidden states remain close to the original input after a small token replacement, while deeper hidden states drift progressively further away. The paper argues this validates the widely used assumption that first-layer representations can stand in for model explanations. It also establishes an empirical regularity relating token frequency to representational sensitivity: frequent tokens such as '.', ',', and numbers yield the smallest embedding shifts, whereas rare tokens yield the largest. The authors attribute this to common tokens carrying less semantic load and having closer semantic neighbors, while rare tokens are underdetermined in the embedding space.

Load-bearing premise

The conclusion rests on treating the $\ell^2$ distance between the hidden states of an input and its minimally perturbed version as a direct measure of how faithfully the layer preserves input information, so the claim that early layers are better explanation proxies collapses if that identification is wrong.

Editorial extensions

If this is right

  • If early layers preserve input information better, interpretability methods that read attributions or nearest-token associations from the first layers inherit that fidelity, while deeper-layer readings should carry a caveat.
  • Rare-token analysis is the sensitive spot: because rare tokens move the representation most, explanations built around them are the ones most likely to change under small input edits.
  • The final layer's deviation from the trend means task-specific layers should be treated as output machinery, not as faithful input encodings.
  • Because all three norms rank the minimally perturbing tokens almost identically, the choice of distance metric is unlikely to change which tokens an explanation pipeline should inspect.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not separate the growing distance between perturbed and original hidden states from the growth in hidden-state scale across layers; normalizing per-layer distances before comparing would test whether the trend reflects information mixing or simply larger activations.
  • A natural extension the authors do not draw is to use the same minimal perturbations to measure causal importance: if rare tokens produce larger embedding shifts, they may also be the tokens whose removal most changes the sentiment prediction, a claim that erasure experiments could verify.
  • The closest-token retrieval used to illustrate deep-layer drift could itself be turned into a quantitative faithfulness metric: computing token-recovery accuracy at each layer and correlating it with the perturbation distance would connect this work directly to the identifiability literature it cites.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a minimal token perturbation framework for probing the embedding space of a BERT model fine-tuned on IMDb sentiment analysis. A token is replaced by its nearest neighbor under cosine similarity in the embedding space, and the resulting shift is measured in the embedding and in the hidden states of successive layers. The experiments report that punctuation and frequent tokens produce minimal shifts, that rare tokens lead to larger embedding shifts, and that the mean L2 distance between original and perturbed hidden states increases with depth except for the final layer. From these observations, the authors argue that input information becomes increasingly intermixed in deeper layers and that early layers are better proxies for model explanations.

Significance. If established, the proposed perturbation-based analysis would be a simple and inexpensive tool for studying information propagation in Transformers, and it would provide direct evidence for the common assumption that early-layer representations are more interpretable. The manuscript is clearly written, the experimental setup is simple enough to reproduce, and the qualitative closest-token evolution example in Section 3.3 is illustrative. However, the central claim is not currently supported by the evidence: the main metric lacks control conditions and normalization, and no independent faithfulness or explanation-quality measure is computed. The contribution is thus at the level of a hypothesis-generating empirical observation rather than a validated interpretability result.

major comments (4)
  1. [§3.3, Figure 3] The increasing mean L2 distance with depth is presented as evidence that 'input information is increasingly intermixed' and that early layers are better proxies for explanations. This inference is not justified without control conditions. A random-token-substitution baseline and a maximal-perturbation baseline are needed to determine whether the trend is specific to minimally perturbed tokens or is a generic property of deep-layer sensitivity to any input change. Additionally, because hidden-state norms grow with depth in BERT-like models, the unnormalized L2 distance can increase even when the relative information change is constant; the paper should report normalized distances or per-layer norm ratios.
  2. [§3.3 and §5] The load-bearing premise is that robustness to perturbations in hidden-state space is equivalent to faithfulness or interpretability. The paper states in Section 3.3 that early layers 'appear to be more robust to perturbations' and then uses this to support their use as explanation proxies, and Section 5 repeats the claim as a validation. This conclusion is read off the same quantity that is being measured. An independent metric is required, such as token identifiability rates following Brunner et al. (2019), output-level attribution, or a probing classifier's performance, to test whether early layers actually provide better explanations.
  3. [§2 and §3.3] The replacement token is selected by minimizing cosine distance in the embedding space, but the propagation analysis in Section 3.3 measures unnormalized L2 distances between hidden states. The relationship between these two metrics is not established; the 'minimal' perturbation in the embedding space need not be the minimal perturbation in terms of L2 displacement at deeper layers. The authors should either justify the compatibility of the two metrics or compare results with perturbations selected by L2 distance in the embedding space.
  4. [§3.2, Figure 2] The commonness–distance regression is the quantitative support for the claim that rare tokens induce larger shifts, but the paper reports only a decreasing trend with a 'tight 95% confidence interval' and does not give the regression slope, intercept, R², or a significance test. The correlations in Section 3.1 are likewise reported without confidence intervals. Please report these statistics and clarify how the binning procedure, including the footnote that some bins contain fewer than 50 unique tokens, affects the regression.
minor comments (7)
  1. [Abstract] The phrase 'the frequency of which tokens yield to minimal shifts' should be 'the frequency with which tokens yield minimal shifts'.
  2. [§3.1] The explanation that frequent tokens such as punctuation and numbers are minimally perturbing is plausible, but it would be strengthened by a control comparison with random frequent-token substitutions rather than only the minimal-perturbation selection.
  3. [§3.2, footnote 2] The binning description is unclear: 'select 50 distinct sentences per bin' is immediately qualified by 'higher commonness bins contain fewer samples.' Please specify exactly how tokens, sentences, and commonness bins are assigned and how the regression weights unequal bin sizes.
  4. [§3.3] The sentence 'the first two tokens lead to smaller shifts... the first two tokens are indeed the most similar in terms of meaning' is ambiguous; it refers to the top-5 least-impactful substitution ranks shown in Figure 3, not to token positions in the input. Please rewrite to avoid confusion.
  5. [§3.3] The closest-token output for the last hidden state contains apparent formatting artifacts such as 'scenerycuit' and 'siskill'; please clean or reformat the example.
  6. [§5] The conclusion states that early layers 'preserve the input more faithfully.' Given the concerns above about the perturbation-distance metric, 'faithfully' should be qualified as 'faithful with respect to the perturbation-distance measure' unless an independent faithfulness evaluation is added.
  7. [General] A statement on code and data availability would improve reproducibility, especially since the experiments are described as runnable in under 10 minutes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's measurements and conclusions are empirical, with no fitted parameter, uniqueness theorem, or self-citation chain doing load-bearing work.

full rationale

The paper's derivation chain is empirical rather than definitional. Section 2 defines a minimal token perturbation by choosing the most cosine-similar replacement token and then measures the resulting shift in the embedding space and in later hidden states. The reported findings, such as rare tokens inducing larger shifts and perturbation distance growing across layers, are direct measurements under that definition. The central claim that first layers can serve as explanation proxies is presented as an interpretation of the increasing mean L2 distance in Figure 3, not as a mathematically derived consequence. There is no fitted parameter that is later renamed as a prediction, no equation that reduces to its own input by construction, and no load-bearing self-citation: the cited works (Brunner et al., Zhang et al., Fantozzi and Naldi) are external, and the paper does not invoke a uniqueness theorem from its own authors. The soft spot identified by a skeptical reader is that 'information preservation' is operationalized as small perturbation distance, so the conclusion partly restates the measurement; however, that is an interpretive validity concern about what the metric means, not a circular derivation in which the output is identical to the input by construction. Under the stated rules, this does not rise to circularity. The paper is therefore self-contained against external benchmarks for the purpose of this circularity analysis.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new fitted model, no new mathematical framework, and no new postulated entity. Its central claims rest on hand-chosen analysis parameters, a fitted descriptive regression, and interpretive assumptions that equate embedding distance with faithfulness and frequency with semantic load. The only genuinely new ingredient is the specific measurement protocol, which is assembled from existing parts.

free parameters (2)
  • Regression slope and intercept for commonness vs embedding distance = not reported
    The decreasing-trend claim in Section 3.2 rests on a fitted linear regression; the slope and intercept are fit to the binned data and no values are given, so the trend cannot be quantitatively checked.
  • Analysis hyperparameters: 10 bins, 50 sentences per bin, 500 sequences, top-5 tokens = 10, 50, 500, 5
    These hand-chosen analysis parameters shape the frequency counts and the regression trend, and no sensitivity analysis is provided.
assumptions (5)
  • standard math All norms are topologically equivalent in finite-dimensional vector spaces.
    Invoked in Section 3.1 to explain why the L1, L2, and L-infinity norms produce highly correlated rankings of minimally perturbing tokens.
  • domain assumption Cosine similarity between token embeddings preserves semantic similarity well enough that nearest-neighbor replacement avoids out-of-distribution inputs.
    Stated in Section 2 and Section 4 as the justification for the perturbation mechanism; no empirical check of semantic preservation is provided.
  • ad hoc to paper Hidden-state distance between an input and its minimally perturbed version measures information preservation and interpretability.
    Section 3.3 interprets the increasing L2 distance as decreasing identifiability and as support for using early layers as explanation proxies. This is the key interpretive assumption, and the paper provides no independent faithfulness metric.
  • domain assumption Token frequency in the IMDb corpus estimates semantic commonness and information load.
    Section 3.2 uses log-transformed corpus frequencies as the commonness score, implicitly assuming frequency correlates with semantic load.
  • domain assumption Findings from one BERT model fine-tuned for IMDb sentiment analysis generalize to Transformers in general.
    The abstract and conclusions speak broadly about Transformers, while the experiments use a single checkpoint and task; the limitation is acknowledged in Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Probing the Embedding Space of Transformers via Minimal Token Perturbations." pith.science (2026). https://pith.science/paper/GXJFG2KC

@misc{pith2026250618011,
  author       = {Pith},
  title        = {Pith review of: Probing the Embedding Space of Transformers via Minimal Token Perturbations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GXJFG2KC}},
  note         = {Machine review of arXiv:2506.18011}
}
read the original abstract

Understanding how information propagates through Transformer models is a key challenge for interpretability. In this work, we study the effects of minimal token perturbations on the embedding space. In our experiments, we analyze the frequency of which tokens yield to minimal shifts, highlighting that rare tokens usually lead to larger shifts. Moreover, we study how perturbations propagate across layers, demonstrating that input information is increasingly intermixed in deeper layers. Our findings validate the common assumption that the first layers of a model can be used as proxies for model explanations. Overall, this work introduces the combination of token perturbations and shifts on the embedding space as a powerful tool for model interpretability.

Figures

Figures reproduced from arXiv: 2506.18011 by the authors.

Figure 1
Figure 1. Tokens most frequently identified as minimally perturbing: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Analysis of embedding distances when perturbing tokens [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The propagation across layers of the top-5 least impactful [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages

  1. [1]

    Is attention explanation? an intro- duction to the debate

    [Bibalet al., 2022 ] Adrien Bibal, R ´emi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang, Thomas Franc ¸ois, and Patrick Watrin. Is attention explanation? an intro- duction to the debate. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3889–3900,

    work page 2022

  2. [4]

    The explainability of transformers: Current status and directions.Computers, 13(4):92,

    [Fantozzi and Naldi, 2024] Paolo Fantozzi and Maurizio Naldi. The explainability of transformers: Current status and directions.Computers, 13(4):92,

    work page 2024

  3. [7]

    [Jain and Wallace, 2019] Sarthak Jain and Byron C

    Association for Computational Linguistics. [Jain and Wallace, 2019] Sarthak Jain and Byron C. Wallace. Attention is not Explanation. InProceedings of the 2019 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, Minneapolis, Minnesota, June

    work page 2019

  4. [8]

    [Khanet al., 2022 ] Salman Khan, Muzammal Naseer, Mu- nawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah

    Associa- tion for Computational Linguistics. [Khanet al., 2022 ] Salman Khan, Muzammal Naseer, Mu- nawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1–41,

    work page 2022

  5. [9]

    Open sesame: Getting inside BERT’s linguistic knowledge.arXiv preprint arXiv:1906.01698,

    [Linet al., 2019 ] Yongjie Lin, Yi Chern Tan, and Robert Frank. Open sesame: Getting inside BERT’s linguistic knowledge.arXiv preprint arXiv:1906.01698,

    arXiv 2019

  6. [10]

    Linguistic interpretability of transformer-based language models: a systematic review

    [L´opez-Otalet al., 2025 ] Miguel L ´opez-Otal, Jorge Gracia, Jordi Bernad, Carlos Bobed, Luc ´ıa Pitarch-Ballesteros, and Emma Angl ´es-Herrero. Linguistic interpretability of transformer-based language models: a systematic review. arXiv preprint arXiv:2504.08001,

    arXiv 2025

  7. [13]

    Word lengths are optimized for efficient communication.Proceedings of the National Academy of Sciences, 108(9):3526–3529,

    [Piantadosiet al., 2011 ] Steven T Piantadosi, Harry Tily, and Edward Gibson. Word lengths are optimized for efficient communication.Proceedings of the National Academy of Sciences, 108(9):3526–3529,

    work page 2011

  8. [14]

    Robinson.An Introduction to Functional Analysis

    [Robinson, 2020] James C. Robinson.An Introduction to Functional Analysis. Cambridge University Press,

    work page 2020

Show all 20 references
  1. [15]

    On linear identifiability of learned representa- tions,

    [Roederet al., 2021 ] Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representa- tions,

    2021

  2. [17]

    Attention is all you need.Advances in neural information processing systems, 30,

    [Vaswaniet al., 2017 ] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30,

    2017

  3. [19]

    [Zhanget al., 2022 ] Chiyuan Zhang, Samy Bengio, and Yoram Singer

    Asso- ciation for Computational Linguistics. [Zhanget al., 2022 ] Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal?Journal of Machine Learning Research, 23(67):1–28,

    2022

  4. [20]

    Explainability for large lan- guage models: A survey.ACM Transactions on Intelligent Systems and Technology, 15(2):1–38, 2024

    [Zhaoet al., 2024 ] Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large lan- guage models: A survey.ACM Transactions on Intelligent Systems and Technology, 15(2):1–38, 2024

    2024

  5. [2011]

    Inter- pretability needs a new paradigm,

    [Madsenet al., 2024 ] Andreas Madsen, Himabindu Lakkaraju, Siva Reddy, and Sarath Chandar. Inter- pretability needs a new paradigm,

    2024

  6. [2017]

    Attention is not not explanation

    [Wiegreffe and Pinter, 2019] Sarah Wiegreffe and Yuval Pin- ter. Attention is not not explanation. InProceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCN...

    2019

  7. [2019]

    BERT: Pre-training of deep bidirectional transformers for language understand- ing

    [Devlinet al., 2019 ] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. InProceedings of the 2019 conference of the North American chapter of the association for computational...

    2019

  8. [2020]

    [Jacovi and Goldberg, 2020] Alon Jacovi and Yoav Gold- berg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? InPro- ceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205, Online, July

    2020

  9. [2021]

    [Schick and Sch¨utze, 2020] Timo Schick and Hinrich Sch¨utze. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking.Pro- ceedings of the AAAI Conference on Artificial Intelligence, 34(05):8766–8774,

    2020

  10. [2022]

    On identifiability in transformers.arXiv preprint arXiv:1908.04211,

    [Brunneret al., 2019 ] Gino Brunner, Yang Liu, Damian Pas- cual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. On identifiability in transformers.arXiv preprint arXiv:1908.04211,

    2019 arXiv

  11. [2024]

    Survey on automatic text summarization and transformer models applicability

    [Guanet al., 2020 ] Wang Guan, Ivan Smetannikov, and Man Tianxing. Survey on automatic text summarization and transformer models applicability. InProceedings of the 2020 1st International Conference on Control, Robotics and Intelligent System, pages 176–184,

    2020

  12. [2025]

    Learning word vectors for sentiment analysis

    [Maaset al., 2011 ] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. InPro- ceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies,...

    2011

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.