The Phasor Transformer: Resolving Attention Bottlenecks on the Unit Circle

Dibakar Sigdel

The Phasor Transformer: Resolving Attention Bottlenecks on the Unit Circle

T0 review · 4 major / 0 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sequence tokens as pure phases on the unit circle, mixed by a fixed DFT and cheap phase shifts, give global coupling without quadratic attention maps.

desk verdict Clean phase-native DFT block with real parameter thrift, but the abstract’s frontier claims outrun the synthetic tables. read the letter →

arxiv 2603.17433 v2 pith:PQVDSCBF submitted 2026-03-18 cs.LG cs.AI

Dibakar Sigdel This is my paper · ORCID

classification cs.LGcs.AI

keywords phasortransformerunitcircleDFTtokenmixingtime-seriesforecastingefficienttransformersphase-nativemodelsLargeModel

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard transformers pay a quadratic cost to mix every token with every other token. This paper argues that for oscillatory time series you can instead put each token on the unit circle as a pure phase, then mix the whole sequence with a parameter-free Discrete Fourier Transform flanked by lightweight trainable phase shifts. Stacking those Phasor Transformer blocks defines a Large Phasor Model whose trainable parameters grow only linearly with context length while still coupling every position to every other at O(N log N) cost. On synthetic multi-frequency autoregressive tasks the model is competitive with—but not better than—self-attention at a small fraction of the parameter count, and deep stacks stay stable when intermediate phases are folded back onto a bounded interval. The central claim is that scalable temporal modeling in periodic domains can come from geometry-constrained phase computation with deterministic global mixing, not only from denser or sparser learned attention.

What carries the argument

The Phasor Transformer block B(θ) = S(θ_post) F_T S(θ_pre): diagonal trainable phase rotations sandwich a fixed Discrete Fourier Transform that couples every token to every other without building an N×N attention matrix; inter-block arcsin(sin(·)) pull-back keeps deep stacks on a bounded phase chart.

What would settle it

Train an LPM and a matched self-attention baseline on a standard long-horizon real benchmark (for example electricity or traffic at contexts of hundreds of steps); if the phasor model loses its claimed parameter advantage without matching competitive error, or if deep stacks diverge once the synthetic fold is removed, the scaling claim fails.

Watch

Extended reading notes

Core claim

Representing sequence states as phases on the unit-circle manifold and mixing them with a parameter-free DFT between trainable pre- and post-phase shifts yields global token interaction without explicit attention maps. Stacked into a Large Phasor Model with inter-block phase pull-back, the architecture learns stable autoregressive dynamics on multi-frequency synthetic series and sits on an explicit efficiency–accuracy frontier: lower error than a zero-parameter baseline, competitive-but-not-superior to self-attention, at far fewer parameters and O(N log N) mixing complexity.

Load-bearing premise

The paper assumes that short synthetic multi-frequency windows and a simple phase-fold between blocks are enough evidence that this design will scale to real long-context time-series foundation models.

Editorial extensions

If this is right

Long-context oscillatory forecasting can use O(N log N) deterministic DFT mixing instead of O(N²) attention maps.
Trainable parameters per block scale as 2T (plus a linear readout) rather than O(d²), so deeper or longer models stay compact.
Inter-block phase pull-back keeps stacked unitary mixers on a bounded manifold, supporting deeper circuits.
Efficiency–accuracy trade-offs for time-series models can be designed around phase geometry, not only sparse or approximate attention.
Domains whose structure is phase or frequency (biosignals, climate, finance, industrial telemetry) become natural targets for phase-native stacks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Hybrid stacks that route some channels through phasor DFT mixing and others through selective learned attention could close the residual accuracy gap without restoring full quadratic cost.
The same unit-circle state plus fixed spectral mixer may transfer to other circular data—angles, relative phases, or periodic spatial fields—beyond the synthetic series tested.
If linear parameter growth holds under longer real contexts, continuous telemetry on edge hardware becomes more practical than dense transformer baselines.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful core is a specific block: unit-circle encoding, trainable pre/post phase gates around a fixed DFT, and arcsin(sin) pull-back between stacks, packaged as LPM. That combination is the new piece. FNet already did Fourier mixing; complex nets already exist. Forcing states onto S¹/T^N with those gates and the inter-block fold, then measuring the parameter–error pairs, is what this paper actually adds.

What it does well is formal and empirical honesty about the design point. Theorem 2.1 is correct: global mixing at O(T log T) with trainable count linear in T and depth. The tables are clear that you get ~50–64 phase parameters versus thousands for a small TransformerEncoderLayer, and the model does learn the synthetic multi-frequency one-step task (test MSE ~0.07, MAE ~0.18). The deep D=3 rollout with pull-back is stable and the loss contracts. Citations cover efficient Transformers, FNet, and time-series variants without obvious gaps. Self-cite to PhasorFlow is library plumbing, not circular proof.

Soft spots are real but bounded. Self-attention is substantially more accurate on the same synthetic data (MSE ~0.003, MAE ~0.10). Calling that “competitive” is a stretch; it is a compactness result. The abstract’s stronger anchors—beating a zero-parameter persistence baseline, and monotonic depth gains under a “corrected gradient path”—are not tabulated or ablated in the body. Contexts stay at T=10–32 on controlled series. No error bars, seeds, or code. That undercuts the “path to long-context temporal foundation models” language more than it undercuts the block itself.

This is for people who care about efficient oscillatory time-series and geometry-constrained mixers, not for general LLM scaling. The math is sound, the data are simple and re-runnable, and the efficiency–accuracy trade-off is visible even if oversold. I would send it to referees after the abstract is aligned with the tables and a real series or persistence ablation is added. Worth a look if you work on spectral or phase-native sequence models; not a must-read otherwise.

Referee Report

4 major / 0 minor

Summary. The paper proposes the Phasor Transformer block B(θ)=S(θ_post) F_T S(θ_pre), which represents sequence tokens as phases on the unit circle S^1 and mixes them with a parameter-free DFT flanked by trainable pre/post phase shifts. Stacking these blocks with an inter-block pull-back Φ_norm=arcsin(sin(ϕ_raw)) defines the Large Phasor Model (LPM). The authors prove linear parameter growth ((2D+1)T) and O(T log T) global mixing (Theorem 2.1), and evaluate one-step and short multi-step autoregressive forecasting on synthetic multi-frequency series (T∈{10,16,32}). Relative to a PyTorch self-attention baseline they report substantially fewer trainable parameters (e.g., 50 vs >1000; 64 vs 3329) at higher error (test MSE ~0.07 vs ~0.003; MAE 0.1817 vs 0.1034), framing this as an explicit efficiency–accuracy frontier for oscillatory sequence modeling.

Significance. If the efficiency–accuracy frontier holds under stronger evaluation, the work would offer a clean, geometry-constrained alternative to dense attention for periodic time series: deterministic global coupling, linear parameter scaling in context length, and an interpretable phase state. Strengths already present include a transparent block definition, correct unitary/DFT complexity statements (Theorem 2.1, Corollary 2.2), and an explicit parameter-scaling comparison (Table 3). The contribution is currently best read as a compact spectral mixer with phase gates rather than a validated path to long-context temporal foundation models; that stronger claim requires the missing baselines, depth ablations, and real-data evidence the abstract advertises.

major comments (4)

Abstract vs. body mismatch on load-bearing empirical pillars. The abstract states that LPM “beats a zero-parameter persistence baseline and, with the corrected gradient path, improves monotonically with depth before saturating.” Sections 4–6 and Tables 1–2 report only self-attention comparisons and a D=3 rollout (Fig. 5); no persistence numbers, no depth–MSE curve, and no “corrected gradient path” ablation appear. These claims are central to the efficiency–accuracy frontier narrative and must either be documented with tables/figures or removed from the abstract and conclusion.
Overstatement of competitiveness relative to reported accuracy gaps. Table 1 gives Phasor test MSE ~0.07 vs self-attention ~0.003 (~23×); Table 2 gives MAE 0.1817 vs 0.1034. The body discussion (§4.1.2, §5, §7) correctly notes a trade-off, but the abstract and framing still call the result “competitive-but-not-superior.” For the frontier claim to be credible, the paper should quantify the accuracy gap explicitly (e.g., relative error ratios, parameter–error Pareto points) and reserve “competitive” for regimes where the gap is small under matched budgets.
Experimental scope is too narrow for the foundation-model scaling claim. All results use short synthetic multi-frequency windows (primarily T=10/16/32, one-step or 20-step rollouts; §3.1, §4–6). There are no error bars, multi-seed statistics, real-world datasets, or long-context regimes. The conclusion’s “path toward scalable temporal modeling” and “long-context temporal foundation models” therefore rest on an untested extrapolation. At minimum, add (i) multi-seed means±std, (ii) a persistence and a linear/spectral baseline, and (iii) at least one public oscillatory benchmark (e.g., electricity, traffic, or a standard multi-horizon suite) before that language is retained.
Inter-block pull-back is asserted without controlled ablation. Proposition 2.2 and §6 motivate Φ_norm=arcsin(sin(ϕ_raw)) as necessary for deep-stack stability, and Fig. 5 shows a working D=3 run. There is no comparison of deep stacks with vs. without pull-back (or vs. alternative re-embeddings such as angle wrapping or modulus renormalization). Because deep composition is part of the LPM definition (Eq. 11, Fig. 2), a short ablation is needed to support the claim that this specific nonlinearity is what enables stable depth scaling.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation chain: architecture properties follow from the block definition, and forecasting results are standard held-out empirical evaluation.

full rationale

This is an empirical architecture paper, not a first-principles derivation that closes on its own inputs. The Phasor block B(θ)=S(θ_post)F_T S(θ_pre) is defined with 2T trainable phases and a parameter-free DFT; Theorem 2.1’s linear parameter count (2D+1)T and O(T log T) mixing are immediate consequences of that design, not fitted quantities re-labeled as predictions. Training uses Adam on phase parameters; test MSE/MAE are measured on held-out synthetic multi-frequency sequences against a self-attention baseline—standard ML practice, not self-definitional or fitted-input circularity. The PhasorFlow citation supplies library primitives and is by a different author; it is not used as a uniqueness theorem or load-bearing proof of forecasting accuracy. FNet-style DFT mixing is cited and adopted, not smuggled as a novel derivation of the same empirical pattern. Gaps between abstract claims (persistence baseline, depth-monotonic gains) and body tables are evidence/completeness issues, not circular reductions. No step reduces a claimed prediction to its own defining fit or to an unverified self-citation chain.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The central efficiency claim rests on standard unitary DFT algebra plus architectural choices (phase-only trainable gates, inter-block fold, synthetic multi-frequency data). Almost all “learning” capacity is in 2T phase angles per block; evaluation quality depends on hand-chosen depths, contexts, optimizer settings, and a synthetic generative process rather than on a derived physical constant. No new particle-like entity is postulated; the invented pieces are architectural operators.

free parameters (5)

pre/post phase-shift vectors θ_pre, θ_post per block
These are the only trainable weights in each block (2T angles); all reported fits are optimizations of these angles on the synthetic task.
Adam learning rate λ=0.05 and 100-epoch budget
Hand-chosen optimization hyperparameters that determine the reported MSE/MAE trajectories (§4.1.1).
phase initialization range [−π/10, π/10]
Ad hoc initialization band for the 50 (or 64/112) trainable angles; not derived.
context lengths T∈{10,16,32}, depth D (e.g. 1 or 3), embedding width of attention baseline (d=16, 4 heads)
Experimental design knobs that define the efficiency–accuracy comparison points in Tables 1–2 and the deep-stack section.
input amplitude-to-phase scale ϕ_t = (x_t / max|x|) · π/2
Encoding constant that forces phases into [−π/2, π/2]; chosen by design, not learned or derived from data statistics beyond max-abs normalization.

assumptions (4)

standard math The unitary DFT F_T provides global token mixing with O(T log T) cost and preserves ℓ2 energy.
Used throughout §2.3–2.4 and Theorem 2.1; standard Fourier analysis.
domain assumption Oscillatory time series are naturally represented by phases on S¹ / T^N rather than unconstrained Euclidean latents.
Stated in Introduction and §2.2 as motivation for the manifold MLPM; not proved, only motivated.
ad hoc to paper Φ_norm = arcsin(sin(ϕ_raw)) re-embeds inter-block states into a bounded phase chart without destroying task-relevant structure.
Introduced in Prop. 2.2 / §2.5 / §6 to stabilize deep stacks; boundedness is true, preservation of useful dynamics is an architectural hypothesis.
domain assumption Synthetic multi-frequency AR series with additive Gaussian noise are a valid primary benchmark for the claimed efficiency–accuracy frontier.
§3.1 and all of §4–6; no real-world dataset is used to support the frontier claim.

invented entities (2)

Phasor Transformer block B(θ)=S(θ_post) F_T S(θ_pre)
purpose: Replace dense attention with phase gates plus parameter-free DFT mixing on the unit circle.
Core architectural object of the paper; independent evidence is only the synthetic experiments here (and related PhasorFlow library citation).
Large Phasor Model (LPM) as depth-D stack with pull-back
purpose: Define a deep sequence model class with linear parameter growth in context length.
Named composition of the blocks; no external validation beyond this manuscript’s synthetic runs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Phasor Transformer: Resolving Attention Bottlenecks on the Unit Circle." pith.science (2026). https://pith.science/paper/PQVDSCBF

@misc{pith2026260317433,
  author       = {Pith},
  title        = {Pith review of: The Phasor Transformer: Resolving Attention Bottlenecks on the Unit Circle},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PQVDSCBF}},
  note         = {Machine review of arXiv:2603.17433}
}

abstract

Transformer models have redefined sequence learning, yet dot-product self-attention introduces a quadratic token-mixing bottleneck for long-context time-series. We introduce the Phasor Transformer block, a phase-native alternative representing sequence states on the unit-circle manifold $S^1$. Each block combines lightweight trainable phase-shifts with parameter-free Discrete Fourier Transform (DFT) token coupling, achieving global $\mathcal{O}(N\log N)$ mixing without explicit attention maps. Stacking these blocks defines the Large Phasor Model (LPM). We validate LPM on autoregressive time-series prediction over synthetic multi-frequency benchmarks against honest baselines: it beats a zero-parameter persistence baseline and, with the corrected gradient path, improves monotonically with depth before saturating, while remaining competitive-but-not-superior to self-attention at a fraction of the parameter count. Our results establish an explicit efficiency--accuracy frontier, showing that scalable temporal modeling in oscillatory domains can emerge from geometry-constrained phase computation with deterministic global coupling.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 7 linked inside Pith

[1]

Peters, and Arman Cohan

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document trans- former.arXiv preprint arXiv:2004.05150, 2020

arXiv 2004
[2]

Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901, 2020

1901
[3]

Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019. 12

arXiv 1904
[4]

Rethinking attention with performers.International Conference on Learning Representa- tions, 2021

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.International Conference on Learning Representa- tions, 2021

2021
[5]

Fu, Stefano Ermon, Atri Rudra, and Christopher Re

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Informa- tion Processing Systems, 35:16344–16359, 2022

2022
[6]

Bert: Pre-training of deep bidirectional transformers for language understanding.Proceedings of NAACL-HLT, pages 4171–4186, 2019

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.Proceedings of NAACL-HLT, pages 4171–4186, 2019

2019
[7]

An image is worth 16x16 words: Transformers for image recognition at scale.Inter- national Conference on Learning Representations, 2021

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.Inter- national Conference on Learning Representations, 2021

2021
[8]

John Wiley & Sons, 2012

Akira Hirose.Complex-Valued Neural Networks: Advances and Applications. John Wiley & Sons, 2012

2012

Show all 23 references

[9]

Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

2022 arXiv
[10]

Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

2001 arXiv
[11]

Fnet: Mixing tokens with fourier transforms.arXiv preprint arXiv:2105.03824, 2021

James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms.arXiv preprint arXiv:2105.03824, 2021

2021 arXiv
[12]

Arik, Nicolas Loeff, and Tomas Pfister

Bryan Lim, Sercan O. Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting.International Journal of Forecasting, 37(4):1748–1764, 2021

2021
[13]

A time series is worth 64 words: Long-term forecasting with transformers.International Conference on Learning Representations, 2023

Yuqi Nie, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers.International Conference on Learning Representations, 2023

2023
[14]

Information Science Reference, 2009

Tohru Nitta.Complex-Valued Neural Networks: Utilizing High-Dimensional Parameters. Information Science Reference, 2009

2009
[15]

Phasorflow: A python library for unit circle based computing.arXiv preprint arXiv:2603.15886, 2026

Vasu Sharma. Phasorflow: A python library for unit circle based computing.arXiv preprint arXiv:2603.15886, 2026

2026 arXiv
[16]

Efficient transformers: A survey.ACM Computing Surveys, 55(6):1–28, 2022

Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey.ACM Computing Surveys, 55(6):1–28, 2022

2022
[17]

Attention is all you need.Advances in neural information processing systems, 30, 2017

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

2017
[18]

Li, Madian Khabsa, Han Fang, and Hao Ma

Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self- attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020. 13

2006 arXiv
[19]

Timesnet: Temporal 2d-variation modeling for general time series analysis.International Conference on Learning Representations, 2023

Haixu Wu, Tao Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis.International Conference on Learning Representations, 2023

2023
[20]

Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.Advances in Neural Information Processing Systems, 34:22419–22430, 2021

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.Advances in Neural Information Processing Systems, 34:22419–22430, 2021

2021
[21]

Big bird: Transformers for longer sequences.Advances in Neural Information Processing Systems, 33:17283–17297, 2020

Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Panupong Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences.Advances in Neural Information Processing Systems, 33:17283–17297, 2020

2020
[22]

Informer: Beyond efficient transformer for long sequence time-series forecasting

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wan- cai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of AAAI, 35(12):11106–11115, 2021

2021
[23]

Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting.International Conference on Machine Learning, pages 27268–27286, 2022

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting.International Conference on Machine Learning, pages 27268–27286, 2022. 14

2022

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.

[1] [1]

Peters, and Arman Cohan

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document trans- former.arXiv preprint arXiv:2004.05150, 2020

arXiv 2004

[2] [2]

Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901, 2020

1901

[3] [3]

Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019. 12

arXiv 1904

[4] [4]

Rethinking attention with performers.International Conference on Learning Representa- tions, 2021

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.International Conference on Learning Representa- tions, 2021

2021

[5] [5]

Fu, Stefano Ermon, Atri Rudra, and Christopher Re

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Informa- tion Processing Systems, 35:16344–16359, 2022

2022

[6] [6]

Bert: Pre-training of deep bidirectional transformers for language understanding.Proceedings of NAACL-HLT, pages 4171–4186, 2019

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.Proceedings of NAACL-HLT, pages 4171–4186, 2019

2019

[7] [7]

An image is worth 16x16 words: Transformers for image recognition at scale.Inter- national Conference on Learning Representations, 2021

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.Inter- national Conference on Learning Representations, 2021

2021

[8] [8]

John Wiley & Sons, 2012

Akira Hirose.Complex-Valued Neural Networks: Advances and Applications. John Wiley & Sons, 2012

2012