Pith. sign in

REVIEW 11 cited by

FNet: Mixing Tokens with Fourier Transforms

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2105.03824 v4 pith:ISLS4SUC submitted 2021-05-09 cs.CL cs.LG

James Lee-Thorp , Joshua Ainslie , Ilya Eckstein , Santiago Ontanon This is my paper · ORCID
classification cs.CLcs.LG
keywords fnetaccuracylengthsfasterinputmodelsstandardtransformer
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the self-attention sublayers with simple linear transformations that "mix" input tokens. These linear mixers, along with standard nonlinearities in feed-forward layers, prove competent at modeling semantic relationships in several text classification tasks. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder with a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE benchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths, our FNet model is significantly faster: when compared to the "efficient" Transformers on the Long Range Arena benchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all sequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint and is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models outperform Transformer counterparts.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer

    cs.LG 2025-06 conditional novelty 7.0 of 10

    Freezing query and key attention weights still lets transformers form induction heads and stay close to standard performance on language modeling, while random static attention (MixiT) fails on in-context tasks but su...

  2. ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Random Fourier Rotations let transformer position encodings approximate any positive-definite attention kernel; ClockRoPE applies this to model daily/weekly routines in sequential recommendation.

  3. FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention

    cs.LG 2026-07 conditional novelty 6.0 of 10

    FFT-based spectral preprocessing of Q/K projections reduces character-level TinyShakespeare validation loss by up to 79% over standard dot-product attention, with gains attributed to phase-coherent global sequence mixing.

  4. Attacking interpretable NLP systems

    cs.CR 2025-07 conditional novelty 6.0 of 10

    AdvChar is a character-level black-box attack that uses interpreter importance scores to pick tokens, fooling NLP classifiers while keeping post-hoc explanations similar.

  5. BeatFormer: Efficient motion-robust remote heart rate estimation through unsupervised spectral zoomed attention filters

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 14.86k-parameter spectral attention model combining Chirp-Z zoom and unsupervised contrastive learning reaches near state-of-the-art cross-dataset heart rate accuracy under motion.

  6. CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling

    cs.CL 2026-04 unverdicted novelty 5.0 of 10

    CAWN mixes sequences via multi-headed complex phasors and O(L) phase accumulation, claiming 2M-token retrieval at constant ~8.72 GB VRAM in a 150M model.

  7. FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression

    cs.CL 2025-07 conditional novelty 5.0 of 10

    FAEDKV compresses the KV cache by keeping the most important Fourier-frequency components of cached keys and values, reporting gains over eviction baselines on LongBench and position-flat retrieval on needle-in-a-hays...

  8. Scaling Attention to Very Long Sequences in Linear Time with Wavelet-Enhanced Random Spectral Attention (WERSA)

    cs.LG 2025-07 reject novelty 5.0 of 10

    WERSA is a linear-complexity attention mechanism combining Haar wavelets with random feature projections, reporting small accuracy gains over baselines but resting on a flawed softmax approximation proof.

  9. The Phasor Transformer: Resolving Attention Bottlenecks on the Unit Circle

    cs.LG 2026-03 conditional novelty 4.5 of 10

    A DFT-plus-phase-shift “Phasor Transformer” stacks into an LPM that forecasts synthetic multi-frequency series with O(N log N) mixing and far fewer parameters than self-attention, but higher error.

  10. WavePhaseNet: A DFT-Based Method for Constructing Semantic Conceptual Hierarchy Structures (SCHS)

    cs.CL 2026-02 reject novelty 4.0 of 10

    WavePhaseNet claims a 24,576-dimensional LLM embedding can be reduced to ~3,000 dimensions via DFT energy thresholds and regularized with graph Laplacians, but the derivation is arithmetically wrong and untested.

  11. Wavelet Logic Machines: Learning and Reasoning in the Spectral Domain Without Neural Networks

    cs.LG 2025-07 reject novelty 4.0 of 10

    The paper claims that a fully spectral wavelet-domain model can reach near-Transformer accuracy on GLUE tasks while using 72% fewer parameters and no attention or convolution layers.

Pith tools