Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Artifacts and Attention Sinks: Structured Approximations for Efficient Vision Transformers

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Structured attention sinks in vision transformers can be exploited for a training-free Nyström approximation that matches full attention while cutting compute.

desk verdict Solid empirical trick, broken headline complexity claim — the FNA forward pass is quadratic in N and the paper admits it in §4.2. read the letter →

arxiv 2507.16018 v1 pith:5EY6ZA44 submitted 2025-07-21 cs.CV

Andrew Lu , Wentinn Liao , Liuhui Wang , Huzheng Yang , Jianbo Shi This is my paper · ORCID
classification cs.CV
keywords attentionsinksmassivetokensartifactNyströmfarthestpointsamplingtraining-freeapproximationvisiontransformersefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision transformers running on images develop a handful of 'massive' tokens whose activation norms explode and which soak up attention like sinks; when those are masked, other 'artifact' tokens grow to replace them. The paper argues that this redundancy is systematic—massive and artifact tokens mutually suppress rival tokens during a distinct emergence phase—and that it can be exploited at inference time without any training. It introduces Fast Nyström Attention, which replaces the quadratic attention matrix with a low-rank Nyström approximation whose landmark tokens are chosen by farthest-point sampling, and shows that on pretrained CLIP and DINOv2 ViT L-14 the approximation matches full attention on retrieval, classification, and segmentation while cutting time and memory. A second, nearly free masking procedure removes the sink tokens in final layers and gives small consistent accuracy gains. If correct, the result turns a quirk of pretrained vision transformers into a drop-in efficiency upgrade.

What carries the argument

The central object is the structured attention pattern formed by massive and artifact tokens: in middle-to-late layers of CLIP and DINOv2, a handful of 'sink' tokens absorb a disproportionate share of attention, and masked-out sinks are replaced by dormant artifact tokens in a roughly fixed priority order. FNA exploits this by writing the attention matrix as a Nyström approximation $A \approx SF(QK^T/\sqrt{d})\,SF(qk^T/\sqrt{d})^{-1}\,SF(qK^T/\sqrt{d})$ with $s$ landmark features, where landmarks are selected by farthest-point sampling (FPS) with the CLS token guaranteed. The mutual-suppression analysis—tokens' value subspaces project negatively onto rivals during the emergence phase—explains why only a few tokens become sinks and why FPS, which spreads sampled points over the feature manifold, naturally captures the outlier sinks. Sampling once after massive tokens form and reusing the landmarks in later layers avoids recomputing FPS per layer.

What would settle it

Run FNA with sample size 64 on a pretrained MAE ViT L-16—a backbone the paper's own appendix shows produces no massive activations—and compare ImageNet linear-probe or retrieval accuracy against full attention; if the gap grows much larger than the sub-half-point gaps reported for CLIP and DINOv2, the assumption that sink tokens are statistical outliers is what carries the method, not the Nyström approximation itself.

Watch

Extended reading notes

Core claim

Fast Nyström Attention (FNA) is a training-free replacement for self-attention in pretrained vision transformers. Standard Nyström attention approximates softmax attention as a product of three softmax matrices built from a small set of landmark tokens; FNA chooses those landmarks by farthest-point sampling over token features, guaranteeing only the CLS token, because massive and artifact tokens are statistical outliers and therefore naturally fall into the sample. With a sample size of 64 on CLIP and DINOv2 ViT L-14, FNA reproduces full-attention results to within a few tenths of a point on COCO and Flickr30k retrieval, ImageNet zero-shot classification, and VOC2012/ADE20k linear-probe segmentation, and it speeds up LLaVA-NeXT-7B VQA generation by about 10% at unchanged BERTScore. The paper also shows that masking detected sink tokens in the final layers consistently improves retrieval and modestly improves classification and segmentation. The stated complexity reduction is from $O(N^2D)$ to $O(sND)$ per attention block in time and from $O(N^2+ND)$ to $O(sN+ND)$ in memory; the farthest-point sampling subroutine itself costs $O(N^2D)$ time and $O(N^2)$ space, though sampling once and reusing landmarks across layers makes total cost $O(N^2D + sLND)$.

Load-bearing premise

The load-bearing premise is that attention-sink tokens are statistical outliers in token-feature space, so farthest-point sampling picks them up automatically; the paper's own appendix shows MAE ViT L-16 has no massive activations, so for that backbone the premise fails and FNA's guarantees are not established.

Editorial extensions

If this is right

  • Pretrained CLIP and DINOv2 ViT L-14 can run with a sample size of 64 landmarks and keep retrieval, classification, and segmentation metrics within about half a point of full attention, with no fine-tuning.
  • A one-time farthest-point sample taken after massive tokens form can be reused across later layers, so the per-layer cost drops to $O(sND)$ after one FPS pass.
  • Masking sink tokens in the final layers improves CLIP COCO text-to-image Recall@1 from 35.33 to 37.47 and gives small consistent gains on ImageNet, VOC2012, and ADE20k.
  • Applied to LLaVA-NeXT-7B, approximating the causal attention over image tokens in the LLM raises generated-token throughput by about 10% without lowering BERTScore.
  • The mutual-suppression account predicts that removing massive tokens early simply promotes artifact tokens, so efficiency must leave the sink population intact; only end-stage masking can safely remove them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication not drawn in the paper: the 'linear time and space' claim applies to the attention module once landmarks are fixed; the full forward pass still includes an $O(N^2D)$ farthest-point pass unless sampling is amortized or replaced by a cheaper landmark detector.
  • Because FNA treats landmark selection as a purely geometric problem on token features, it should transfer to any attention-based model whose activations exhibit the same outlier structure, including multimodal and long-sequence settings; the paper only demonstrates CLIP, DINOv2, and one LLaVA variant.
  • The artifact-token redundancy suggests a direct test of register-trained models: if registers reduce massive tokens, FNA's sample may no longer find them, so the method may need to guarantee register or sink tokens explicitly.
  • Masking gains hint at a training-free denoising pass that could be combined with FNA: if sink tokens are excluded from the landmark set as well as from final-layer attention, the approximation and the denoising might compound.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies massive tokens (attention sinks) and artifact tokens in pretrained vision transformers, argues that these tokens mutually suppress one another through attention, and introduces Fast Nyström Attention (FNA), a training-free method that approximates self-attention using Nyström extension with farthest point sampling (FPS). It also proposes masking sink tokens in later layers to denoise features. Experiments on CLIP and DINOv2 ViT-L14 cover zero-shot retrieval, classification, segmentation, and LLaVA-based VQA, reporting competitive performance with reduced memory and time; additional ablations compare sampling strategies, sample sizes, and finetuned efficient-attention baselines.

Significance. If the empirical claims hold, FNA would be a practical drop-in efficiency upgrade for CLIP/DINOv2-style backbones, and the masking analysis would connect to existing attention-sink phenomena in a useful way. The paper's strengths are its systematic ablation of all 27 sampling configurations, the sample-size sweep, the finetuned comparison with Linformer and Performer, and the LLaVA extension. The paper does not provide code and reports no repeated-run statistics, and the headline complexity claim is contradicted by the paper's own complexity analysis in Section 4.2. The theoretical mechanism in Section 7 is plausible but largely qualitative. As an empirical systems contribution the result is potentially significant, but the current presentation overstates the efficiency and scope claims.

major comments (3)
  1. [Section 4.2 and abstract] The paper's own complexity accounting states that the FPS subroutine requires O(N^2D) time and O(N^2) space and that the total forward pass costs O(N^2D + sLND) in time with peak memory O(N^2 + ND); only the attention block after landmarks are fixed is O(sND). The abstract and introduction nevertheless claim that FNA approximates self-attention 'in linear time and space' and reduces computational overhead. These statements are internally inconsistent. The efficiency claim should be corrected to the honest bound, and Table 1's timings should clarify whether FPS overhead is included, since the 256-token row (FNA 1.2 ms vs standard attention 0.8 ms) is otherwise unexplained.
  2. [Tables 2, 3, 5, and 7] The performance-preservation claim rests on point estimates without error bars, repeated runs, or significance tests. For example, Table 2 shows CLIP+FNA+resample COCO image R@1 of 35.58 versus 35.33 baseline, and Table 3 shows ImageNet top-1 of 75.81 versus 75.96; these differences are small enough that run-to-run variability could change the conclusions. The manuscript should add variance estimates or bootstrap intervals and specify the evaluation protocol (number of seeds, any stochasticity in FPS initialization, and whether the same sample is reused across images).
  3. [Section 4.2 and Appendix C.3] The sampling justification assumes that massive and artifact tokens are statistical outliers on the feature manifold so that FPS naturally represents the sink-token population. Appendix C.3 explicitly shows that MAE ViT L-16 does not produce massive activations, and the paper does not test FNA on MAE or on any other backbone without sink tokens. The title and abstract refer to 'vision transformers' generally, so the scope is overclaimed. Either evaluate a no-sink backbone and characterize the failure mode, or explicitly restrict the claims to models with established sink-token structure such as CLIP and DINOv2.
minor comments (4)
  1. [Appendix B.1, Definition B.2] The notation 'SFsink(w,m)= md SF(w)' is undefined; it appears to mean elementwise multiplication of the softmax output by the mask, but this should be stated explicitly.
  2. [Appendix B.1] The sentence 'For operand tensors of multiple dimensions, these operations similarly to SF will only be relevant on the last dimension' is grammatically incomplete and should be rephrased.
  3. [Figure 11] The reported baselines (43.28% text-to-image R@1 and 51.44% image-to-text R@1) do not match Table 5's CLIP COCO baselines (56.06% text-to-image R@1 and 35.33% image-to-text R@1); please clarify the evaluation setting or correct the labels.
  4. [Section 6 and Appendix A.4] The masking experiments do not state whether they use the iterative Algorithm 1 or the non-iterative spectral clustering from Section 3.4; if iterative detection is used, the claim that masking is 'at virtually no cost' requires a wall-clock breakdown of the multiple forward passes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FNA's derivation reuses an externally attributed Nyström approximation, and its performance and efficiency claims are empirical rather than forced by construction.

full rationale

I found no circular step that reduces the paper's claims to its own inputs. The core approximation in Eq. (6) is explicitly attributed to Xiong et al. [25], an external source, and FNA contributes a change in landmark selection (FPS) plus a one-time sampling strategy; the resulting attention approximation is then measured against external benchmarks (COCO, Flickr30k, ImageNet, VOC2012, ADE20k, VQA). The performance claims are therefore empirical, not derived from the method's assumptions. Hyperparameters such as the sample size of 64 and detection layers are reported as ablations or grid searches (Appendix A.1, Section 3.2), not disguised as predictions. The assumption that massive and artifact tokens are statistical outliers that FPS will naturally capture is an empirical premise, tested in Figure 8 and Appendix A.1, and is not an identity. The self-citations [20, 26] are used for spectral clustering and NCut visualization; they are minor references and are not load-bearing for the central efficiency argument. The paper's own complexity analysis in Section 4.2 admits that the FPS subroutine requires O(N^2D) time and O(N^2) space, which contradicts the abstract's 'linear time and space' phrasing, but this is an internal correctness or overclaiming issue rather than a circular derivation. No equation is equivalent to its inputs by construction, and no fitted parameter is renamed as a prediction. The honest finding is therefore no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central method rests on a small number of tuned hyperparameters, all disclosed in appendices, and on assumptions about the low-rank, outlier structure of ViT attention. The heaviest assumption is the outlier-manifold property for FPS, which is validated only for models that exhibit massive activations.

free parameters (4)
  • FNA sample size s = 64 (swept from 1 to 128 in Fig. 9; 16-512 in LLaVA)
    Number of landmark tokens used in the Nyström approximation; chosen by grid search on COCO retrieval.
  • Detection layer (ℓ_detection) = 13 for CLIP ViT L-14, 20 for DINOv2 ViT L-14
    Layer where token identities are read off via the CLS-attention threshold; selected based on observed massive-token formation.
  • Masking layers = final layers (Figure 1 shows last two layers; Section 6 says final layers)
    Which later layers have sink tokens masked and replaced with nearest normal tokens; the ablation in Fig. 11 shows the effects vary by layer.
  • Sink threshold = A_CLS->t >= A_CLS->CLS
    Threshold used in Algorithm 1 to call a token an attention sink; a hand-set criterion, though the paper shows it captures known massive tokens.
assumptions (4)
  • domain assumption Nyström approximation of the softmax attention matrix (Eq. 6) is accurate at inference time using a small landmark set.
    The method inherits this from Xiong et al. and assumes the landmark set represents the column and row space of the attention matrix well enough that downstream tasks are preserved.
  • domain assumption Attention matrix in middle-to-late ViT layers is low-rank due to sink tokens.
    Motivates the whole FNA approach; supported by Figure 2 but not proven.
  • domain assumption Farthest point sampling selects massive and artifact tokens because they are feature-manifold outliers.
    Section 4.2 states this directly; the paper's own ablation (Fig. 8) supports it empirically for CLIP, and Appendix C.3 shows it fails to apply when no massive activations exist (MAE).
  • standard math MLP nonlinearities and asymptotic Lipschitzness bound token growth.
    Used in Appendix B.3.1 to explain why lower-ranked tokens saturate during iterative masking; stated informally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Artifacts and Attention Sinks: Structured Approximations for Efficient Vision Transformers." pith.science (2026). https://pith.science/paper/5EY6ZA44

@misc{pith2026250716018,
  author       = {Pith},
  title        = {Pith review of: Artifacts and Attention Sinks: Structured Approximations for Efficient Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5EY6ZA44}},
  note         = {Machine review of arXiv:2507.16018}
}
read the original abstract

Vision transformers have emerged as a powerful tool across a wide range of applications, yet their inner workings remain only partially understood. In this work, we examine the phenomenon of massive tokens - tokens with exceptionally high activation norms that act as attention sinks - and artifact tokens that emerge as a byproduct during inference. Our analysis reveals that these tokens mutually suppress one another through the attention mechanism, playing a critical role in regulating information flow within the network. Leveraging these insights, we introduce Fast Nystr\"om Attention (FNA), a training-free method that approximates self-attention in linear time and space by exploiting the structured patterns formed by massive and artifact tokens. Additionally, we propose a masking strategy to mitigate noise from these tokens, yielding modest performance gains at virtually no cost. We evaluate our approach on popular pretrained vision backbones and demonstrate competitive performance on retrieval, classification, segmentation, and visual question answering (VQA), all while reducing computational overhead.

Figures

Figures reproduced from arXiv: 2507.16018 by the authors.

Figure 1
Figure 1. Visualizations of sink (MA + Artifact) tokens masks applied to image features in the last two layers of CLIP [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Each subplot visualizes the mean attention matrix across heads for a single image at a particular layer of CLIP ViT L-14. In an [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Each subplot depicts the result of a step in the iterative removal process for a single image in CLIP ViT L-14. The leftmost attention [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Fast Nyström Attention outperforms existing linear atten [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: BERTScore [31] and generation speed on COCO VQA [1] for different configurations of Fast Nyström Attention (FNA) applied to LLaVA-NeXT-7B [15]. Each color represents a LLaVA model with FNA applied on the causal attention to image tokens at the specified span of layers.…
Figure 6
Figure 6. Figure 6: Fitting a linear probe to the average token grouped by type [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: We observe in layers 9 and 10 that the pairwise suppression within the set of (potential) sink tokens is particularly strong. This [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: COCO retrieval metrics on all [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Furthest point sampling (FPS) sample size vs. perfor [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Example outputs generated by LLaVA-NeXT-7B [ [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 12
Figure 12. Figure 12: Plot of activation norms of tokens across 50 images over [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Different masking patterns with respect to the interest set [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 15
Figure 15. Figure 15: Running the model while ignoring the attention mecha [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]
Figure 14
Figure 14. Figure 14: The x-axis shows individual tokens that are ranked by order of removal in the iterative masking procedure, with the same sequence of top 20 tokens showed for all subplots. The y-axis of subfigure 14a denotes the (unsunk) attention from the CLS token which we use as a …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    Layer-wise Sink Gating scales vision and LLM attention sinks in LVLMs to balance global priors and local evidence, improving multimodal benchmarks with a frozen backbone.

  2. Activation Quantization of Vision Encoders Needs Prefixing Registers

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Inserting precomputed universal register tokens in the middle layers of pretrained vision encoders shrinks activation outliers, and deleting emerging sink tokens, improves low-bit post-training quantization accuracy.

Reference graph

Works this paper leans on

37 extracted references · 29 canonical work pages · cited by 2 Pith papers

  1. [1]

    Lawrence Zitnick, Dhruv Batra, and Devi Parikh

    Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. VQA: Visual Question Answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2425–2433, 2015. 6, 7, 11, 12

    work page 2015

  2. [2]

    Peters, and Arman Cohan

    Iman Beltagy, Matthew E. Peters, and Arman Cohan. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. 2

    arXiv 2004

  3. [3]

    Lawrence Zitnick

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data and evaluation server for im- age captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4565–4574. IEEE, 2015. 1, 5, 6, 7, 11

    work page 2015

  4. [4]

    Learn- ing a sparse transformer network for effective image deraining

    Xiang Chen, Hao Li, Mingqiang Li, and Jinshan Pan. Learn- ing a sparse transformer network for effective image deraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE/CVF, 2023. 2

    work page 2023

  5. [5]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. 2

    arXiv 1904

  6. [6]

    Performer: Linear attention via positive random features

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xiang Song, Adrian Gane, Thomas Sarlos, Peter Hawkins, James Davis, Adam Weller, Sam Gardner, et al. Performer: Linear attention via positive random features. arXiv preprint arXiv:2009.14794v4, 2020. 2, 6, 11

    arXiv 2009

  7. [7]

    Flashattention-2: Faster attention with better paral- lelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better paral- lelism and work partitioning. In Proceedings of the 12th In- ternational Conference on Learning Representations (ICLR),

    work page

  8. [8]

    Vision transformers need registers

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bo- janowski. Vision transformers need registers. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. 2

    work page 2024

Show all 37 references
  1. [9]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE, 2009. 1, 5, 6, 13

    2009

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  3. [11]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http:// www . pascal - network . org / challenges / VOC / voc2012/workshop/index.html, 2012. 1, 5, 6, 13

    2012

  4. [12]

    When attention sink emerges in language models: An empirical view

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410.10781, 2024. 1, 2

    2024 arXiv

  5. [13]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009, 2022. 1

    2022

  6. [14]

    Openclip, July

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Ha- jishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July

  7. [15]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023. 6, 7, 11, 12

    2023

  8. [16]

    Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas...

  9. [17]

    Qi, Li Yi, Hao Su, and Leonidas J

    Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Point- net++: Deep hierarchical feature learning on point sets in a metric space, 2017. 5

    2017

  10. [18]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference...

    2021

  11. [19]

    Combiner: Full attention transformer with sparse computation cost

    Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, and Bo Dai. Combiner: Full attention transformer with sparse computation cost. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021. 2

    2021

  12. [20]

    Normalized cuts and image segmentation

    Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8):888–905, Aug 2000. 4

    2000

  13. [21]

    Zico Kolter, and Zhuang Liu

    Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762, 2024. 2

    2024 arXiv

  14. [22]

    Deit iii: Re- venge of the vit

    Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Re- venge of the vit. In Proceedings of the European Conference on Computer Vision (ECCV), 2022. 1

    2022

  15. [23]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), pages 5998–6008,

  16. [24]

    Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and W. Ma. Linformer: Self-attention with linear complexity. In International Conference on Learning Representations (ICLR), 2020. 2, 6, 11

    2020

  17. [25]

    Nys- trömformer: A nyström-based algorithm for approximating self-attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nys- trömformer: A nyström-based algorithm for approximating self-attention. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 35(1):1111–1119, 2021. 2, 4, 5, 6, 11 9

    2021

  18. [26]

    Ncut apis – nyström normalized cuts py- torch

    Huzheng Yang. Ncut apis – nyström normalized cuts py- torch. https://ncut-pytorch.readthedocs.io/ en/latest/api_reference/, 2024. Accessed: 2025- 03-04. 2, 4

    2024

  19. [27]

    Emernerf: Emergent spatial-temporal scene decomposition via self-supervision

    Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Se- ung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fi- dler, Marco Pavone, and Yue Wang. Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. In International Conference on Learning Representatio...

    2024

  20. [28]

    Denoising vision transformers

    Jiawei Yang, Katie Z Luo, Jiefeng Li, Congyue Deng, Leonidas Guibas, Dilip Krishnan, Kilian Q Weinberger, Yon- glong Tian, and Yue Wang. Denoising vision transformers. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. 2

    2024

  21. [29]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

    Peter Young, Alice Lai, Michael Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. In Transactions of the Association for Computational Linguistics (TACL), volume 2, pages 67–...

    2014

  22. [30]

    The super weight in large language models

    Mengxia Yu, De Wang, Qi Shan, Colorado Reed, and Alvin Wan. The super weight in large language models. arXiv preprint arXiv:2411.07191, 2024. 2

    2024 arXiv

  23. [31]

    Wein- berger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Wein- berger, and Yoav Artzi. BERTScore: Evaluating Text Gener- ation with BERT. In International Conference on Learning Representations (ICLR), 2020. 6, 7

    2020

  24. [32]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Anto- nio Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 633–641. IEEE, 2017. 1, 5, 6, 13 10 A. Additional R...

    2017

  25. [34]

    Type I sinking set T

    is performed via fitting linear probes to output of the final layers. We show minor but consistent performance gains from masking sink tokens in the final layers. A.5. Masking Gains on Classification and Segmen- tation Section 6 demonstrates how masking out massive and artifac...

  26. [35]

    removing the attention mechanism in layers 9, 10, 11, 12 result in some artifact tokens becoming massive that are not massive in the unmodified computational path, and

  27. [36]

    retain their place

    for any interest set T (includingH) that Type I mask- ing or sinking in layers 9, 10, 11, 12 elicits the same set of massive tokens as Type I masking or sinking in layers 9, 10 and proceeding without attention in layers 11 and 12. This suggests that while the MLP in layers 11 ...

  28. [37]

    On the other hand, the attention pattern for any token t‰ t1 is identical to that of Type I sinking

    Because the attention pattern of t1 itself is untouched by Type II sinking t1 alone, its value at the intermediate output of layer 9 is identical to that of unmodified computation. On the other hand, the attention pattern for any token t‰ t1 is identical to that of Type I sink...

  29. [2021]

    If you use this software, please cite it as below. 5

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.