Pith. sign in

REVIEW 4 cited by

A Survey of Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2106.04554 v2 pith:KWMGKDEG submitted 2021-06-08 cs.LG cs.AIcs.CL

Tianyang Lin , Yuxin Wang , Xiangyang Liu , Xipeng Qiu This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords x-formerstransformercomprehensivegreatintroducenaturalprocessingreview
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Atleus: Accelerating Transformers on the Edge Enabled by 3D Heterogeneous Manycore Architectures

    cs.AR 2025-01 conditional novelty 7.0 of 10

    A 3D heterogeneous ReRAM-plus-systolic-array accelerator that claims up to 56x speedup and 64.5x energy efficiency over GPUs for transformer fine-tuning and inference.

  2. Toward Manifest Relationality in Transformers via Symmetry Reduction

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Transformer attention and parameter optimization can be rewritten on symmetry-reduced relational variables (Gram matrices and invariant parameter composites), removing coordinate redundancies by construction.

  3. Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictions

    cs.LG 2025-06 reject novelty 6.0 of 10

    SurvBESA applies self-attention to predicted survival functions from bagged Beran estimators and reports improved ranking performance on benchmark survival datasets.

  4. Tracking UWB Devices Through Radio Frequency Fingerprinting Is Possible

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Deep learning can extract stable device fingerprints from UWB signals, achieving over 99% accuracy in fixed conditions and above-chance performance in untrained environments.

Pith tools