Pith. sign in

REVIEW 5 major objections 5 minor 39 references

Wavelet Logic Machines: Learning and Reasoning in the Spectral Domain Without Neural Networks

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a model with no convolution, attention, or MLP layers—only learnable wavelet-domain operations—achieves near-Transformer accuracy on SST-2 with 72 percent fewer parameters.

desk verdict A genuinely interesting spectral-learning architecture, but the GLUE results are internally inconsistent, unreproducible, and rest on an unspecified token layout; desk reject rather than referee. read the letter →

arxiv 2507.19514 v1 pith:2XSGNL7R submitted 2025-07-18 cs.LG

Andrew Kiruluta This is my paper · ORCID
classification cs.LG
keywords wavelettransformspectrallearningsoft-thresholdinglearnablebasisselectionGLUEbenchmarkSST-2sentimentclassificationmodelefficiencyinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that learning does not require neural network layers: a model that works only on wavelet coefficients, with learnable shrinkage, gain-phase modulation, and a softmax choice among wavelet bases, can denoise 3D signals and classify natural-language sentences. On GLUE tasks the model reports accuracy near 90 percent on SST-2, close to a four-layer Transformer (90.1 percent) while using 72 percent fewer parameters and 58 percent less peak memory. The authors claim faster early convergence from the spectral sparsity prior and $O(n)$ instead of $O(n^2)$ complexity for long inputs. If this holds, a compact, inspectable, spectral model is a viable alternative to overparameterized neural architectures for at least some vision and language workloads.

What carries the argument

The load-bearing object is the 3D discrete wavelet transform (DWT) and its inverse, used as the only feature-extraction step. Around it sit three learnable mechanisms: the pointwise spectral nonlinearity $\phi(z) = \gamma\, \mathrm{sign}(z)\,\max(|z|-\lambda,0)\,\cos\theta$ with learnable threshold $\lambda$, gain $\gamma$, and phase $\theta$; a softmax over learnable logits $\alpha_k$ that mixes reconstruction paths from candidate wavelet families (Haar, Daubechies, Symlet, and Biorthogonal); and an entropy penalty that prunes low-weight bases. The argument is that these operations, applied subband by subband, supply the filtering and feature selection that neural networks normally learn with spatial filters.

What would settle it

Re-run the reported GLUE experiments from the recipe in Section 5: fixed pretrained token embeddings, 3D reshape, wavelet soft-thresholding, and no positional encodings. If SST-2 accuracy does not land near 90 percent, or if the 0.7M-parameter count and the claimed memory savings do not reproduce, the paper's central quantitative claim is unsupported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a complete learning pipeline can be built from wavelet-domain operations alone: the input is decomposed by a discrete wavelet transform, the coefficients are modified by learnable soft-thresholding and gain-phase modulation, and the inverse transform reconstructs a representation that is either a denoised signal or the basis for a classifier. The model has no convolution, attention, or MLP layers, and no positional encodings; the only trainable quantities are thresholds $\lambda$, gains $\gamma$, phase shifts $\theta$, and softmax basis-selection logits $\alpha_k$. The author reports that this 0.7M-parameter spectral model reaches roughly 90 percent accuracy on SST-2 and competitive scores on MRPC and QNLI, close to a four-layer Transformer, using 72 percent fewer parameters and 58 percent less peak memory. The intended lesson is that spectral sparsity and multiscale structure carry much of the inductive bias that deep neural architectures are normally assumed to provide.

Load-bearing premise

The load-bearing premise is that reshaping a token sequence into a 3D tensor and processing it with wavelets, without any positional encoding, preserves the word-order and phrase-level information needed for sentiment and inference classification.

Editorial extensions

If this is right

  • A linear-time wavelet pipeline would remove the $O(n^2)$ attention bottleneck for long sequences, lowering inference cost on large inputs.
  • If the GLUE numbers hold, a 0.7M-parameter model would come within a few points of the large pretrained transformer baseline on SST-2, MRPC, and QNLI while being over 150x smaller.
  • Learned thresholds and gains give a per-subband record of what the model suppresses or amplifies, enabling frequency-localized explanations for predictions.
  • The differentiable basis selector lets the model choose different wavelet families per task, adapting its inductive bias without manual filter design.
  • The spectral sparsity prior could speed early training convergence, reducing compute before full convergence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's Section 9 reasoning extensions are presented as proposals without experiments; a concrete next step is running them on a small reasoning benchmark to see whether spectral cascades or keyed memory improve accuracy beyond the base pipeline.
  • If the 3D reshape is truly order-insensitive, the SST-2 result would suggest sentiment classification is largely decodable from local frequency statistics of token embeddings; shuffling token order before reshaping would test this directly.
  • The abstract reports 89.3 percent while Table 1 reports 90.1 percent for the same model, and memory savings are given as both 58 percent and 'over 70 percent' in different passages, so an exact reproducible run is needed to pin down the quantitative claim.
  • Because the language pipeline starts from fixed pretrained embeddings, part of the result may be inherited from those embeddings; replacing them with learned or contextual embeddings would isolate the contribution of the wavelet mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes 'Wavelet Logic Machines,' a spectral learning framework that replaces convolutions, attention, and MLPs with a pipeline of 3D discrete wavelet transforms, learnable soft-thresholding, gain-phase modulation, and differentiable wavelet-basis selection. The main empirical claim is that this ~0.7M-parameter model reaches near-Transformer accuracy on GLUE tasks (89.3% or 90.1% on SST-2 depending on the passage) while using far fewer parameters and less memory. The paper also argues for interpretability and sketches extensions for symbolic reasoning, including spectral rule composition, cascades, a DSL, and keyed memory access.

Significance. If substantiated, the central claim would be significant: a parameter-efficient model without attention or convolution that approaches BERT-base on SST-2, MRPC, and QNLI would challenge standard assumptions about what language classification requires. The differentiable basis-selection mechanism and the linear-time complexity argument are genuinely interesting components. However, the paper currently provides no reproducible evidence for the GLUE results, and the empirical core is internally inconsistent. The strengths of the proposal cannot compensate for the absence of a verifiable experimental record.

major comments (5)
  1. [Abstract and Table 1] The headline numbers are internally inconsistent. The abstract reports 89.3% SST-2 accuracy against a 4-layer Transformer baseline of 90.1%, with 72% fewer parameters and 58% less peak memory. Table 1 instead reports the Wavelet-Spectral model at 90.1% SST-2, lists BERT-base (93.5%) rather than any 4-layer Transformer, and Section 5 later claims peak memory reduction 'over 70%.' These are not minor wording differences; the paper cannot support two different flagship accuracy numbers and two different memory-savings figures for the same experiment. The authors must reconcile the abstract, the results table, and the efficiency text.
  2. [Section 5] The language representation is not specified at a load-bearing level. The text says each token sequence is 'reshaped into a three-dimensional tensor of shape (C, H, W)' with 'H × W determined by reshaping or padding,' but there is no layout diagram, no padding or truncation rule, no description of how sentence pairs are packed for MRPC and QNLI, and no positional encoding. Because the only token-to-token interactions are fixed local wavelet filters followed by global pooling and a linear head, it is not established that word order and phrase structure survive this representation. Without a layout specification, an ablation over layouts, or code, the reported SST-2, MRPC, and QNLI accuracies are unsupported.
  3. [Sections 3.2 and 5] The classification forward pass is undefined. The model in Section 3 is a denoiser: it applies DWT, soft-thresholding, gain-phase modulation, basis selection, IDWT, and minimizes reconstruction MSE. Section 5 states that the final output is pooled and passed to a single linear layer with a cross-entropy loss, but no equation or architecture for this classification path is given, and there is no explanation of how the denoising parameters (λA, λD, γ, θ, α) relate to the classifier. This makes the GLUE results impossible to check against the model definition.
  4. [Section 5 and Table 1] The experimental protocol is missing. The paper gives no dataset splits, tokenizer, embedding dimension, sequence length, batch size, learning rate, number of epochs, or number of runs for any GLUE result. The baselines are said to be trained 'under identical conditions,' but the BERT-base fine-tuning details are absent, and no code is provided. Given that a 0.7M-parameter model without contextual layers is claimed to outperform a 2.1M-parameter BiLSTM on these benchmarks, the lack of reproducibility evidence is a load-bearing gap.
  5. [Sections 6, 7, and 8] The interpretability and reasoning claims are not backed by the experiments. Section 6 asserts that in GLUE experiments the model assigns high probability to Daubechies-4 and Symlet-6 bases, and Section 7 claims that Level 1 might isolate negations, but no quantitative analysis, learned-weight inspection, or visualization from the actual GLUE runs is provided. Section 8 analyzes synthetic sentence embeddings with the Fourier transform, not the trained model's wavelet representations. These sections therefore do not support the paper's advertised interpretability and reasoning contributions.
minor comments (5)
  1. [Table 1] There are typographical errors: 'W avelet-Spectral' and 'BER T-base' should be 'Wavelet-Spectral' and 'BERT-base.'
  2. [References] The reference list contains duplicates (e.g., [23]/[8], [24]/[9], [26]/[11], [34]/[20]) and inconsistent numbering; please deduplicate and renumber.
  3. [Section 5] The text says the model 'applies 2D or 3D DWT' while the rest of the paper claims full 3D support; the GLUE experiments should specify which transform is actually used.
  4. [Title and Abstract] The phrase 'Without Neural Networks' is not consistent with the final linear classifier and pretrained embeddings; the paper should clarify that it eliminates hidden neural layers, not all learned linear components.
  5. [Section 9] The five reasoning modules are speculative and are not evaluated on any task; consider moving this material to future work or adding experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular dependency: the model's GLUE accuracies are trained and evaluated externally, and the cited prior work is related-work positioning only.

full rationale

The paper's derivation chain is a set of differentiable wavelet-domain operations (DWT, soft-thresholding, gain-phase modulation, softmax basis weighting, IDWT) trained by MSE or cross-entropy and evaluated on held-out GLUE benchmarks. The reported SST-2/MRPC/QNLI numbers are external benchmark results, not quantities reconstructed from fitted parameters: the learnable parameters (lambda, gamma, theta, alpha) are optimized on training losses and then measured on test sets, so no fitted input is renamed as a prediction. The method's equations are stated in the paper itself (Sec. 3.1-3.2, Sec. 4), so the central claim does not reduce to a citation. The self-citations [15,16,17] are used only to describe related spectral-dictionary, wavelet-attention, and memory-based models; they are not invoked as authority for the current model's correctness, and no 'uniqueness theorem' or ansatz is imported from those papers. The unspecified 3D reshape of token embeddings and the absence of positional encodings are substantive reproducibility and validity concerns, but they are not definitional circularity: the claimed accuracies are not equal to the training loss by construction. The abstract/table discrepancy (89.3% vs 90.1% SST-2) is an internal inconsistency, not a circular step. Accordingly, no circularity is found.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce physical or mathematical entities such as new particles, forces, or dimensions. The main added objects are trainable parameters listed above and proposed algorithmic modules in Section 9, which are not realized or evaluated, so they are not counted as independently evidenced entities.

free parameters (7)
  • lambda_A (approximation soft-threshold) = not reported
    Learnable threshold in the nonlinearity phi(z) = gamma * sign(z) * max(|z|-lambda,0) * cos(theta); controls sparsity of approximation coefficients.
  • lambda_D (detail soft-threshold) = not reported
    Same learnable nonlinearity applied to detail coefficients; learned by gradient descent on the reconstruction or classification loss.
  • gamma (amplitude gain) = not reported
    Positive learnable scale applied to all wavelet coefficients; described as global or channel-wise.
  • theta (modulation term) = not reported
    Learnable scalar multiplied as cos(theta); called phase modulation, but it is not an actual complex phase because coefficients are real.
  • alpha (basis selection logits) = not reported
    K-dimensional logits normalized by softmax to combine reconstructions from different wavelet bases; these are global constants, not per-input parameters.
  • beta (entropy regularization weight) = not reported
    Hand-chosen strength of the basis sparsity penalty in the total loss L_total = L_recon + beta * L_entropy.
  • Td, smax, tau, lambda_prune (schedule and pruning hyperparameters) = not reported
    Introduced in Section 3.3 for the wavelet dilation schedule and basis pruning; no values or ablations are reported.
assumptions (4)
  • standard math DWT/IDWT with orthogonal or biorthogonal wavelets satisfies perfect reconstruction.
    Invoked in Sections 2 and 3.2 through pywt.dwtn and pywt.idwtn; assumed true for the wavelets chosen by PyWavelets.
  • domain assumption Reshaped GloVe or FastText embedding tensors preserve task-relevant information for classification without positional encoding.
    Section 5 assumes token sequences reshaped into (C,H,W) tensors can be classified via wavelet shrinkage; no positional encoding is used and the assumption is not validated.
  • ad hoc to paper Global softmax basis weights can be interpreted as symbolic or semantic selection.
    Sections 3.1 and 6 claim the basis weights expose latent inductive bias, but the weights are input-independent logits, so the interpretation is not supported by the architecture alone.
  • domain assumption Wavelet subbands map onto linguistic features such as negation and sentence structure.
    Sections 7 and 8 claim Level 1 isolates abrupt negations and Level 3 captures global structure, but no experiment measures this mapping.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Wavelet Logic Machines: Learning and Reasoning in the Spectral Domain Without Neural Networks." pith.science (2026). https://pith.science/paper/2XSGNL7R

@misc{pith2026250719514,
  author       = {Pith},
  title        = {Pith review of: Wavelet Logic Machines: Learning and Reasoning in the Spectral Domain Without Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2XSGNL7R}},
  note         = {Machine review of arXiv:2507.19514}
}
read the original abstract

We introduce a fully spectral learning framework that eliminates traditional neural layers by operating entirely in the wavelet domain. The model applies learnable nonlinear transformations, including soft-thresholding and gain-phase modulation, directly to wavelet coefficients. It also includes a differentiable wavelet basis selection mechanism, enabling adaptive processing using families such as Haar, Daubechies, and Biorthogonal wavelets. Implemented in PyTorch with full 3D support, the model maintains a spectral pipeline without spatial convolutions or attention. On synthetic 3D denoising and natural language tasks from the GLUE benchmark, including SST-2 sentiment classification, the model achieves 89.3 percent accuracy, close to a 4-layer Transformer baseline (90.1 percent), while using 72 percent fewer parameters and 58 percent less peak memory. Faster early convergence is observed due to spectral sparsity priors. In contrast to the quadratic complexity of self-attention and large matrix multiplications in Transformers, our approach uses linear-time wavelet transforms and pointwise nonlinearities, significantly reducing inference cost. This yields a compact, interpretable, and efficient alternative to neural models. Our results support the viability of principled spectral learning in both vision and language tasks, offering new directions for model design without overparameterized architectures.

Figures

Figures reproduced from arXiv: 2507.19514 by the authors.

Figure 1
Figure 1. Architecture of the fully spectral wavelet-based model for volumetric denoising. Given [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Wavelet detail coefficients of synthetic sentence embeddings using Daubechies-2 (db2) [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 33 canonical work pages

  1. [1]

    LeCun, L

    Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    work page 1998

  2. [2]

    Hochreiter and J

    S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735– 1780, 1997

    1997

  3. [3]

    J., and Adelson, E

    Burt, P. J., and Adelson, E. H. The Laplacian Pyramid as a Compact Image Code. IEEE Transactions on Communications, 31(4), 532–540, 1983

    work page 1983

  4. [4]

    K., Vandergheynst, P., and Gribonval, R

    Hammond, D. K., Vandergheynst, P., and Gribonval, R. Wavelets on Graphs via Spectral Graph Theory. Applied and Computational Harmonic Analysis, 30(2), 129–150, 2011

    work page 2011

  5. [5]

    Feature Detection with Automatic Scale Selection

    Lindeberg, T. Feature Detection with Automatic Scale Selection. International Journal of Computer Vision, 30(2), 77–116, 1998

    work page 1998

  6. [6]

    A Wavelet Tour of Signal Processing

    Mallat, S. A Wavelet Tour of Signal Processing. Academic Press, 1999

    work page 1999

  7. [7]

    I., Narang, S

    Shuman, D. I., Narang, S. K., Frossard, P., Ortega, A., and Vandergheynst, P. The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains. IEEE Signal Processing Magazine, 30(3), 83–98, 2013

    work page 2013

  8. [10]

    Bruna and S

    J. Bruna and S. Mallat. Invariant scattering convolution networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1872–1886, 2013. 19

    work page 2013

Show all 39 references
  1. [11]

    A. Wang, Y. Pruksachatkun, N. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations, 2019

    2019

  2. [13]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  3. [14]

    Sweldens

    W. Sweldens. The lifting scheme: A construction of second generation wavelets. SIAM Journal on Mathematical Analysis, 29(2):511–546, 1996

    1996

  4. [15]

    From Attention to Atoms: Spec- tral Dictionary Learning for Fast, Interpretable Language Models

    Kiruluta, Andrew, Burity, Priscilla, and Williams, Samantha. From Attention to Atoms: Spec- tral Dictionary Learning for Fast, Interpretable Language Models. CoRR, abs/2505.00033, 2025

    2025 arXiv

  5. [16]

    Learnable Multi-Scale Wavelet Transformer: A Novel Alternative to Self-Attention

    Kiruluta, Andrew, Burity, Priscilla, and Williams, Samantha. Learnable Multi-Scale Wavelet Transformer: A Novel Alternative to Self-Attention. CoRR, abs/2504.03821, 2025

    2025 arXiv

  6. [17]

    Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons

    Kiruluta, Andrew, Lundy, Eric, and Burity, Priscilla. Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons. CoRR, abs/2506.01963, 2025

    2025 arXiv

  7. [18]

    Pennington, R

    J. Pennington, R. Socher, and C. Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543, 2014

    2014

  8. [19]

    Devlin, M

    J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019

    2019

  9. [21]

    S. Mallat. A theory for multiresolution signal decomposition: The wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7):674–693, 1989

    1989

  10. [23]

    Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anand- kumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020

    2010 arXiv

  11. [24]

    Lee-Thorp, J

    J. Lee-Thorp, J. Ainslie, I. Eisner, and S. Ontanon. FNet: Mixing tokens with Fourier trans- forms. arXiv preprint arXiv:2105.03824, 2021

    2021 arXiv

  12. [25]

    S. Lu, B. Zheng, X. Wu, and S. Zhou. Fourierformer: Transformer meets generalized Fourier transform. arXiv preprint arXiv:2106.15137, 2021. 20

    2021 arXiv

  13. [26]

    A. Wang, Y. Pruksachatkun, N. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. beginthebibliography10

    2019

  14. [27]

    S. Mallat. A theory for multiresolution signal decomposition: the wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7):674–693, 1989

    1989

  15. [30]

    D. L. Donoho and I. M. Johnstone. Adapting to unknown smoothness via wavelet shrinkage. Journal of the American Statistical Association, 90(432):1200–1224, 1995

    1995

  16. [32]

    S. Mallat. A Wavelet Tour of Signal Processing. Academic Press, 1999

    1999

  17. [33]

    Zhang, S

    C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In Proceedings of the International Conference on Learning Representations (ICLR), 2017

    2017

  18. [34]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, pages 5998–6008, 2017

    2017

  19. [37]

    Vetterli and J

    M. Vetterli and J. Kovaˇ cevi´ c.Wavelets and Subband Coding. Prentice Hall, 1995

    1995

  20. [38]

    Daubechies

    I. Daubechies. Ten Lectures on Wavelets. SIAM, 1992

    1992

  21. [39]

    Strang and T

    G. Strang and T. Nguyen. Wavelets and Filter Banks. Wellesley-Cambridge Press, 1996

    1996

  22. [40]

    G. Lee, E. P. Simoncelli, and M. S. Lewicki. PyWavelets: A Python library for wavelet transform computations. https://github.com/PyWavelets/pywt, 2006

    2006

  23. [41]

    Cohen, I

    A. Cohen, I. Daubechies, and J. C. Feauveau. Biorthogonal bases of compactly supported wavelets. Communications on Pure and Applied Mathematics, 45(5):485–560, 1993

    1993

  24. [42]

    F. B. Cotter. Pytorch-wavelets: A PyTorch package for wavelet transforms. https://github.com/fbcotter/pytorch wavelets, 2019

    2019

  25. [43]

    Vincent, H

    P. Vincent, H. Larochelle, Y. Bengio, and P. A. Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning (ICML), pages 1096–1103, 2008

    2008

  26. [44]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015. 21

    2015

  27. [45]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, Jacob, Chang, Ming-Wei, Lee, Kenton, and Toutanova, Kristina. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019

    2019

  28. [46]

    RoBERTa: A Robustly Optimized BERT Pretraining Approach

    Liu, Yinhan, Ott, Myle, Goyal, Naman, Du, Jingfei, Joshi, Mandar, Chen, Danqi, Levy, Omer, Lewis, Mike, Zettlemoyer, Luke, and Stoyanov, Veselin. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  29. [47]

    What Does BERT Look At? An Analysis of BERT’s Attention

    Clark, Kevin, Khandelwal, Urvashi, Levy, Omer, and Manning, Christopher D. What Does BERT Look At? An Analysis of BERT’s Attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019

    2019

  30. [48]

    Analyzing the Structure of Attention in a Transformer Language Model

    Vig, Jesse. Analyzing the Structure of Attention in a Transformer Language Model. In Proceedings of the 2019 ACL Workshop on BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019

    2019

  31. [49]

    Axiomatic Attribution for Deep Net- works

    Sundararajan, Mukund, Taly, Ankur, and Yan, Qiqi. Axiomatic Attribution for Deep Net- works. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017. 22

    2017

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.