Pith. sign in

REVIEW 9 cited by

Focal Self-attention for Local-Global Interactions in Vision Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2107.00641 v1 pith:ZEM5FB3P submitted 2021-07-01 cs.CV cs.AIcs.LG

Jianwei Yang , Chunyuan Li , Pengchuan Zhang , Xiyang Dai , Bin Xiao , Lu Yuan , Jianfeng Gao This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords focalvisiontransformerself-attentiontransformersdetectionobjecttasks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability of capturing short- and long-range visual dependencies through self-attention is arguably the main source for the success. But it also brings challenges due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection). In this paper, we present focal self-attention, a new mechanism that incorporates both fine-grained local and coarse-grained global interactions. Using this new mechanism, each token attends the closest surrounding tokens at fine granularity but the tokens far away at coarse granularity, and thus can capture both short- and long-range visual dependencies efficiently and effectively. With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers on a range of public image classification and object detection benchmarks. In particular, our Focal Transformer models with a moderate size of 51.1M and a larger size of 89.8M achieve 83.5 and 83.8 Top-1 accuracy, respectively, on ImageNet classification at 224x224 resolution. Using Focal Transformers as the backbones, we obtain consistent and substantial improvements over the current state-of-the-art Swin Transformers for 6 different object detection methods trained with standard 1x and 3x schedules. Our largest Focal Transformer yields 58.7/58.9 box mAPs and 50.9/51.3 mask mAPs on COCO mini-val/test-dev, and 55.4 mIoU on ADE20K for semantic segmentation, creating new SoTA on three of the most challenging computer vision tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Partial Ring Scan: Revisiting Scan Order in Vision State Space Models

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Ring-based scanning with selective channel routing improves accuracy, speed, and rotation robustness of vision state-space models.

  2. Vision encoders should be image size agnostic and task driven

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A position paper proposing task-driven, image-size-agnostic vision encoders, demonstrated by an iterative transformer with top-down multi-zoom patches and a GRPO-trained gaze policy on ImageNet.

  3. Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A token-space SVD-based method that separately resolves gradient conflicts in the range and null spaces of transformer tokens improves multi-task learning performance with minimal extra parameters.

  4. ViT-Split: Unleashing the Power of Vision Foundation Models via Efficient Splitting Heads

    cs.CV 2025-06 conditional novelty 6.0 of 10

    ViT-Split freezes a vision foundation model and adds a copied task head plus a multi-scale prior head, matching or beating prior adapters with fewer parameters and up to 4x faster training.

  5. MMA-Former: Multi-Window Mixture-of-Head Attention Transformer for Adaptive PNI Prediction in 3D MRI

    cs.CV 2026-07 conditional novelty 5.5 of 10

    MMA-Former with window-level Mixture-of-Head attention reaches AUC 0.752 for PNI prediction on 168 cropped 3D MRI scans, beating 3D CNN and Swin baselines.

  6. Quantum-Enhanced Optimization by Warm Starts

    quant-ph 2025-08 unverdicted novelty 5.0 of 10

    Quantum-generated QAOA samples, used as warm starts for classical heuristics, yield faster Max-Cut and MIS solving, including on quantum hardware.

  7. RelayFormer: A Unified Local-Global Attention Framework for Scalable Image and Video Manipulation Localization

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    RelayFormer splits inputs into fixed-size pieces and uses relay tokens to share global context, aiming for unified image and video tamper localization at any resolution.

  8. A Retrospective Systematic Study on Hierarchical Sparse Query Transformer-assisted Ultrasound Screening for Early Hepatocellular Carcinoma

    cs.CV 2025-02 conditional novelty 5.0 of 10

    A hybrid CNN-transformer with mixture-of-experts (HSQformer) yields 95.38% AUC for HCC screening on a multi-center ultrasound test set, beating several baselines and matching senior radiologists.

  9. AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A vision transformer reduces self-attention complexity from O(n^2) to O(mn) by using m learnable anchor tokens and a two-step Markov transition between anchors and tokens.

Pith tools