Pith. sign in

REVIEW 4 major objections 5 minor 36 references

AttentionSmithy: A Modular Framework for Rapid Transformer Development and Customization

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A modular framework cuts transformer customization overhead and finds that combining all four positional encodings maximizes translation BLEU.

desk verdict A useful, honestly-reported modular transformer framework with public code and two replications, but the 'combined positional encodings' NAS finding is under-supported and over-sold in the abstract. read the letter →

arxiv 2502.09503 v2 pith:XX7VDCKM submitted 2025-02-13 cs.LG cs.AI

Caleb Cranney , Jesse G. Meyer This is my paper · ORCID
classification cs.LGcs.AI
keywords transformerarchitecturesmodularsoftwaredesignpositionalencodingstrategiesneuralarchitecturesearchmachinetranslationsingle-celltranscriptomicsGeneformerbuildingblocks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AttentionSmithy is a software package built to make transformer architectures easy to reassemble: attention, feed-forward networks, normalization layers, and positional encodings are separated into reusable modules with stable interfaces. The paper claims this lowers the implementation barrier for domain experts, and it supports the claim by replicating the original transformer under limited hardware, by running a neural architecture search on a German-to-English task, and by reproducing a gene-expression model that exceeds 95% accuracy on cell-type classification. The headline empirical claim is that the search found machine translation quality is best when all four supported positional encoding strategies—sinusoidal, learned, rotary, and ALiBi—are applied at once, which the paper interprets as evidence that the strategies capture complementary positional information. If correct, the package turns transformer design into assembly of tested components and opens a concrete research lead about combining position-encoding mechanisms.

What carries the argument

The load-bearing mechanism is the positional-embedding strategy manager: a software pattern that lets sinusoidal, learned, rotary, and ALiBi encodings be activated or deactivated independently without touching the core attention or feed-forward code. Around it, the framework provides class-level modules for multi-head attention, feed-forward networks, normalization plus residual sublayer units, encoder/decoder layers, and generators, with interfaces designed so components can be swapped programmatically. This modularity is what makes the neural architecture search possible: a search script toggles the four encodings, dropout, and activation, and a Bayesian optimizer scores configurations by BLEU.

What would settle it

Train the optimized configuration with all four encodings, with each encoding removed one at a time, and with all encodings present but original dropout and ReLU restored; use multiple seeds and 40 epochs. If no encoding removal hurts BLEU, or if restoring dropout and ReLU reproduces the gain without all encodings, then the claimed combined-encoding benefit is not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that transformer components can be decomposed into interchangeable building blocks—multi-head attention, feed-forward networks, sublayer units, encoders/decoders, and a positional-embedding strategy manager—and that this decomposition makes both manual prototyping and automated architecture search practical. Under resource constraints (one A100 GPU, 100-token context window), a vanilla replication reached BLEU approximately 21 on WMT14 English-German after 40 epochs, while the NAS-optimized variant reached BLEU approximately 24, approaching the original paper's 25. The search's best configuration used all four positional encoding methods simultaneously, removed dropout, and switched ReLU to tanh; the authors single out the combined positional encodings as the unexpected and interesting result. In a separate domain, a BERT-style model built with the framework and pretrained on rank-ordered gene tokens achieved over 95% accuracy on cell-type classification, demonstrating that the same modules transfer to genomics.

Load-bearing premise

The paper's combined-encoding result rests on one search run whose candidates trained for five epochs, and the winning configuration changed dropout and activation at the same time, so the BLEU gain could be noise or caused by those other changes rather than by stacking encodings.

Editorial extensions

If this is right

  • Researchers can compare or combine positional encodings with a single configuration change instead of forking the model code.
  • Neural architecture search over transformer components becomes accessible from the same modular code the model is built with.
  • The joint use of all four encodings is presented as a new finding to test on other translation pairs and sequence tasks.
  • The same framework reproduces a published genomics model with over 95% validation accuracy, indicating the building blocks generalize beyond text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the framework lets encodings compose independently at initialization, a researcher could stack explicit temporal and positional information, a combination the paper mentions but does not demonstrate.
  • The modular decomposition suggests architecture search could eventually treat attention sparsity, normalization placement, and residual connections as swappable ingredients, not just encodings.
  • If combined encodings hold up, the default advice for new ordered-data tasks may shift from picking one encoding to searching over combinations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AttentionSmithy, a PyTorch-based modular framework for building transformer architectures from reusable components: attention modules, feed-forward networks, normalization/residual sublayers, and positional encodings. The framework supports four positional encoding strategies (sinusoidal, learned, rotary, and ALiBi) and integrates with neural architecture search (NAS). The authors report three validation studies: a resource-constrained replication of the original Vaswani et al. transformer on WMT14 English-German (BLEU ~21 vs. the original ~25), a NAS-optimized variant achieving ~24 BLEU by combining all four positional encodings while also removing dropout and switching from ReLU to tanh, and a Geneformer-style model for cell type classification that achieves over 95% validation accuracy. The central claims are that the package lowers the barrier to transformer customization and that the NAS run produced an unexpected discovery that combining all positional encoding methods maximizes translation performance.

Significance. If the framework is reliable, it is a useful contribution: it is publicly available under the MIT license, provides clean modular abstractions, and the two replication studies (vanilla transformer and Geneformer) demonstrate that the package can instantiate established architectures and achieve reasonable performance. The framework's design goals—rapid prototyping, educational clarity, and NAS compatibility—are valuable for practitioners. However, the headline empirical finding, the 'unexpected discovery' about combined positional encodings, is not currently supported by the evidence presented. The optimized configuration changes three architectural variables simultaneously, is based on single training trajectories with no seeds or confidence intervals, and the NAS ranking is performed at five epochs while the final result is reported after forty. The Geneformer replication also lacks a direct comparison to the original model's published performance. These issues limit the scientific significance of the empirical claims, though the software contribution itself remains potentially useful.

major comments (4)
  1. [Results, Figure 2A; Methods, Neural Architecture Search Compatibility] The paper attributes the optimized model's BLEU improvement to combining all four positional encodings, but the optimized configuration differs from the vanilla baseline in three jointly varied dimensions: all four positional encodings are enabled, dropout is reduced from 0.1 to 0.0, and the feed-forward activation is changed from ReLU to tanh. Figure 2A shows a single trajectory per condition with no seeds, error bars, or confidence intervals, so the approximately 3-point BLEU gain after 40 epochs could be noise or could be caused by the dropout or activation changes rather than by the positional-encoding combination. In addition, the NAS search evaluates candidates at only 5 epochs, while the reported BLEU scores are after 40 epochs; no evidence is given that the 5-epoch ranking transfers to longer training. Please provide ablations that isolate the positional-encoding combination (for example, all four encodings with dropout=0.1 and ReLU, dropout=0.0 and tanh without the positional-encoding change, and the full optimized configuration), run multiple seeds, and report means with confidence intervals.
  2. [Abstract and Results, Original Transformer Replication] The abstract's phrase 'near state-of-the-art performance' is not supported by the reported numbers. The original transformer paper reported BLEU 25 on WMT14 English-German under different training conditions, while the replication here achieves approximately 21 under a 100-token context window and a single A100 GPU; the optimized model reaches approximately 24. Without error bars, a statistical comparison, or reference to other modern baselines on the same setup, 'near state-of-the-art' is an overstatement. Please rephrase this claim to describe the result as a resource-constrained reproduction and report the actual BLEU values with uncertainty estimates.
  3. [Discussion, Efficacy of Combined Positional Encodings] The 'unexpected discovery' language in the Discussion overstates the strength of the evidence. The paper reports a single NAS run and a single retained configuration; there is no demonstration that the all-four-positional-encoding configuration is robust across seeds, training lengths, or neighboring hyperparameter settings. A search outcome is not a validated empirical finding unless the confounded variables are disentangled and the result is shown to replicate. Please temper the discussion to describe the result as an initial lead from a small search, or provide the additional experiments needed to support the discovery claim.
  4. [Results, Bioinformatics Application] The Geneformer validation reports only that the AttentionSmithy implementation achieved over 95% validation accuracy on the human_dcm_hcm_nf dataset. Since the original Geneformer paper reports classification performance on the same dataset, the replication claim requires a direct side-by-side comparison (for example, original Geneformer accuracy versus the AttentionSmithy implementation) and details on preprocessing, fine-tuning hyperparameters, and the number of training steps. Without these, 'successful replication' is not fully demonstrated, and the reader cannot tell whether the 95% figure is comparable to the original model's performance.
minor comments (5)
  1. [Figure 1A] There is a typo in the figure caption and label: 'Vaswanii' should be 'Vaswani'.
  2. [Methods, Neural Architecture Search Compatibility] The NAS search space is described only in prose. A table listing the six adjustable parameters, their ranges, and the number of trials evaluated would greatly improve reproducibility and clarity.
  3. [Results, Original Transformer Replication] The paper mentions that 63,227 sentences were truncated because they exceeded the 100-token context window, but it does not report the fraction of the training set this represents or how truncation was applied (prefix, suffix, or random selection). This detail matters for interpreting the BLEU difference relative to the original transformer.
  4. [Code Availability] The three GitHub repositories are listed, but no version numbers, commit hashes, or dependency/environment specifications are provided. Including these would make the software contribution more reproducible.
  5. [Results, Bioinformatics Application] The fine-tuning procedure for the Geneformer replication should be specified more precisely, including the number of epochs, learning rate, batch size, and whether the reported accuracy is evaluated on the full validation set.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are a software-engineering contribution and empirical benchmark results, not derivations from their own inputs.

full rationale

AttentionSmithy's claims are validated empirically against external results: the vanilla transformer replication reports BLEU ~21 versus the original paper's 25 under tighter resource constraints, and the Geneformer replication reports >95% cell-type classification accuracy, matching the published task. No load-bearing premise is justified by a self-citation chain or an imported uniqueness theorem. The NAS outcome is an optimization result: the search space included six adjustable parameters with BLEU as the objective, so reporting the best found configuration is a search outcome, not a fitted quantity renamed as a prediction. The attribution of the improvement specifically to combining all four positional encodings is weakened by simultaneous dropout and activation changes and by the absence of multiple seeds, but confounding and missing error bars are robustness concerns rather than circularity. The paper also transparently states its resource constraints and the short 5-epoch NAS training. No equation, definition, or cited result reduces by construction to the paper's own inputs, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or theoretical entities are introduced; the strategy manager is a software abstraction, not an invented entity. The free parameters listed are the empirical choices made by the NAS and resource constraints, while the axioms are the domain assumptions the framework and validation rest on.

free parameters (4)
  • dropout rate in optimized model = 0.0
    Selected by NAS on WMT14 BLEU; changed from the original transformer's 0.1.
  • feed-forward activation in optimized model = tanh
    Selected by NAS; replaced ReLU.
  • context window truncation = 100 tokens
    Chosen due to resource constraints; 63,227 sentences exceeding 100 tokens were truncated.
  • search training length = 5 epochs
    Used for NAS candidate evaluation to reduce time; final models trained for 40 epochs.
assumptions (5)
  • domain assumption Transformer components decompose cleanly into attention, feed-forward, normalization, residual, and positional encoding modules.
    The framework's modular design assumes these are the right separable units; stated in Methods and Figure 1.
  • domain assumption BLEU score is an adequate proxy for translation quality for architecture search.
    Used as the NAS objective; cited from the original transformer paper.
  • ad hoc to paper Five-epoch training during NAS ranks architectures similarly to longer training.
    No evidence is provided that short-run BLEU correlates with 40-epoch BLEU; this underpins the combined-encoding result.
  • domain assumption The Geneformer pretraining and fine-tuning protocol transfers to this re-implementation.
    The paper follows the published methodology and data, but does not report hyperparameters.
  • standard math Softmax attention as defined in Vaswani et al. is the correct base computation.
    Inherited from the cited transformer literature and implemented without modification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AttentionSmithy: A Modular Framework for Rapid Transformer Development and Customization." pith.science (2026). https://pith.science/paper/XX7VDCKM

@misc{pith2026250209503,
  author       = {Pith},
  title        = {Pith review of: AttentionSmithy: A Modular Framework for Rapid Transformer Development and Customization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XX7VDCKM}},
  note         = {Machine review of arXiv:2502.09503}
}
read the original abstract

Transformer architectures have transformed AI applications but remain complex to customize for domain experts lacking low-level implementation expertise. We introduce AttentionSmithy, a modular software package that simplifies transformer innovation by breaking down key components into reusable building blocks: attention modules, feed-forward networks, normalization layers, and positional encodings. Users can rapidly prototype and evaluate transformer variants without extensive coding. Our framework supports four positional encoding strategies and integrates with neural architecture search for automated design. We validate AttentionSmithy by replicating the original transformer under resource constraints and optimizing translation performance by combining positional encodings. Additionally, we demonstrate its adaptability in gene-specific modeling, achieving over 95% accuracy in cell type classification. These case studies highlight AttentionSmithy's potential to accelerate research across diverse fields by removing framework implementation barriers.

Figures

Figures reproduced from arXiv: 2502.09503 by the authors.

Figure 1
Figure 1. Components of the transformer model architecture as coded in AttentionSmithy [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Validation studies of programs built with AttentionSmithy. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 14 canonical work pages

  1. [1]

    Attention Is All You Need,

    A. Vaswani et al., “Attention Is All You Need,” Aug. 02, 2023, arXiv: arXiv:1706.03762. doi: 10.48550/arXiv.1706.03762

    arXiv doi:10.48550/arxiv.1706.03762 2023

  2. [2]

    Exploring ChatGPT and its impact on society,

    Md. A. Haque and S. Li, “Exploring ChatGPT and its impact on society,” AI Ethics, Feb. 2024, doi: 10.1007/s43681-024-00435-4

    work page doi:10.1007/s43681-024-00435-4 2024

  3. [3]

    Long Short-Term Memory,

    S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Comput, vol. 9, no. 8, pp. 1735–1780, Nov. 1997, doi: 10.1162/neco.1997.9.8.1735

    doi:10.1162/neco.1997.9.8.1735 1997

  4. [4]

    Transformers in the Real World: A Survey on NLP Applications,

    N. Patwardhan, S. Marrone, and C. Sansone, “Transformers in the Real World: A Survey on NLP Applications,” Information, vol. 14, no. 4, Art. no. 4, Apr. 2023, doi: 10.3390/info14040242

    doi:10.3390/info14040242 2023

  5. [5]

    A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships,

    G. A. Pereira and M. Hussain, “A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships,” Aug. 27, 2024, arXiv: arXiv:2408.15178. doi: 10.48550/arXiv.2408.15178

    arXiv doi:10.48550/arxiv.2408.15178 2024

  6. [6]

    Transformers in Healthcare: A Survey,

    S. Nerella et al., “Transformers in Healthcare: A Survey,” Artif. Intell. Med., vol. 154, p. 102900, Aug. 2024, doi: 10.1016/j.artmed.2024.102900

    work page arXiv 2024

  7. [7]

    Transformer technology in molecular science,

    J. Jiang et al., “Transformer technology in molecular science,” WIREs Comput. Mol. Sci., vol. 14, no. 4, p. e1725, 2024, doi: 10.1002/wcms.1725

    doi:10.1002/wcms.1725 2024

  8. [8]

    Transformer Architecture and Attention Mechanisms in Genome Data Analysis: A Comprehensive Review,

    S. R. Choi and M. Lee, “Transformer Architecture and Attention Mechanisms in Genome Data Analysis: A Comprehensive Review,” Biology, vol. 12, no. 7, Art. no. 7, Jul. 2023, doi: 10.3390/biology12071033

    doi:10.3390/biology12071033 2023

Show all 36 references
  1. [9]

    What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding,

    Y.-A. Wang and Y.-N. Chen, “What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding,” Oct. 10, 2020, arXiv: arXiv:2010.04903. doi: 10.48550/arXiv.2010.04903

    2020 doi arXiv

  2. [10]

    RoFormer: Enhanced Transformer with Rotary Position Embedding,

    J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu, “RoFormer: Enhanced Transformer with Rotary Position Embedding,” Nov. 08, 2023, arXiv: arXiv:2104.09864. doi: 10.48550/arXiv.2104.09864

    2023 doi arXiv

  3. [11]

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation,

    O. Press, N. A. Smith, and M. Lewis, “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation,” Apr. 22, 2022, arXiv: arXiv:2108.12409. doi: 10.48550/arXiv.2108.12409

    2022 doi arXiv

  4. [12]

    Neural Architecture Search for Transformers: A Survey,

    K. T. Chitty-Venkata, M. Emani, V. Vishwanath, and A. K. Somani, “Neural Architecture Search for Transformers: A Survey,” IEEE Access, vol. 10, pp. 108374–108412, 2022, doi: 10.1109/ACCESS.2022.3212767

    2022

  5. [13]

    Neural Architecture Search on Efficient Transformers and Beyond,

    Z. Liu et al., “Neural Architecture Search on Efficient Transformers and Beyond,” Jul. 28, 2022, arXiv: arXiv:2207.13955. doi: 10.48550/arXiv.2207.13955

    2022 doi arXiv

  6. [14]

    HuggingFace’s Transformers: State-of-the-art Natural Language Processing,

    T. Wolf et al., “HuggingFace’s Transformers: State-of-the-art Natural Language Processing,” Jul. 14, 2020, arXiv: arXiv:1910.03771. doi: 10.48550/arXiv.1910.03771

    2020 doi arXiv

  7. [15]

    PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation,

    J. Ansel et al., “PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation,” in 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24), ACM, Apr. 2024. doi...

    2024

  8. [16]

    Transformer — PyTorch 2.6 documentation

    “Transformer — PyTorch 2.6 documentation.” Accessed: Jan. 31, 2025. [Online]. Available: https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html#torch.nn.Transformer

    2025

  9. [17]

    TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems

    Martín Abadi et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.” 2015. [Online]. Available: https://www.tensorflow.org/

    2015

  10. [18]

    models/official/nlp/modeling at master · tensorflow/models,

    “models/official/nlp/modeling at master · tensorflow/models,” GitHub. Accessed: Jan. 31,

  11. [19]

    Customizing a Transformer Encoder | Text,

    “Customizing a Transformer Encoder | Text,” TensorFlow. Accessed: Jan. 31, 2025. [Online]. Available: https://www.tensorflow.org/tfmodels/nlp/customize_encoder

    2025

  12. [20]

    Ousterhout, A Philosophy of Software Design, 1st ed

    J. Ousterhout, A Philosophy of Software Design, 1st ed. 2018

    2018

  13. [21]

    Vogel, I

    O. Vogel, I. Arnold, A. Chughtai, and T. Kehrer, Software Architecture: A Comprehensive Framework and Guide for Practitioners. Springer Publishing Company, Incorporated, 2011

    2011

  14. [22]

    Gamma, R

    E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design patterns: elements of reusable object-oriented software. USA: Addison-Wesley Longman Publishing Co., Inc., 1995

    1995

  15. [23]

    Transformer models: an introduction and catalog,

    X. Amatriain, A. Sankar, J. Bing, P. K. Bodigutla, T. J. Hazen, and M. Kazi, “Transformer models: an introduction and catalog,” Mar. 31, 2024, arXiv: arXiv:2302.07730. doi: 10.48550/arXiv.2302.07730

    2024 doi arXiv

  16. [24]

    Longformer: The Long-Document Transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The Long-Document Transformer,” Dec. 02, 2020, arXiv: arXiv:2004.05150. doi: 10.48550/arXiv.2004.05150

    2020 doi arXiv

  17. [25]

    Big Bird: Transformers for Longer Sequences,

    M. Zaheer et al., “Big Bird: Transformers for Longer Sequences,” Jan. 08, 2021, arXiv: arXiv:2007.14062. doi: 10.48550/arXiv.2007.14062

    2021 doi arXiv

  18. [26]

    Multi-Objective NAS with Ax — PyTorch Tutorials 2.3.0+cu121 documentation

    “Multi-Objective NAS with Ax — PyTorch Tutorials 2.3.0+cu121 documentation.” Accessed: May 17, 2024. [Online]. Available: https://pytorch.org/tutorials/intermediate/ax_multiobjective_nas_tutorial.html

    2024

  19. [27]

    facebook/Ax. (Feb. 12, 2025). Python. Meta. Accessed: Feb. 12, 2025. [Online]. Available: https://github.com/facebook/Ax

    2025

  20. [28]

    Bleu: a Method for Automatic Evaluation of Machine Translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a Method for Automatic Evaluation of Machine Translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, P. Isabelle, E. Charniak, and D. Lin, Eds., Philadelphia, Pennsylvania,...

    2002

  21. [29]

    Findings of the 2014 Workshop on Statistical Machine Translation,

    O. Bojar et al., “Findings of the 2014 Workshop on Statistical Machine Translation,” in Proceedings of the Ninth Workshop on Statistical Machine Translation, O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, C. Monz, M. Post, and L. Specia, Eds., Baltimore, Maryland, USA: ...

    2014 doi

  22. [30]

    Transfer learning enables predictions in network biology,

    C. V. Theodoris et al., “Transfer learning enables predictions in network biology,” Nature, vol. 618, no. 7965, pp. 616–624, Jun. 2023, doi: 10.1038/s41586-023-06139-9

    2023 doi

  23. [31]

    The Annotated Transformer

    “The Annotated Transformer.” Accessed: Nov. 18, 2024. [Online]. Available: https://nlp.seas.harvard.edu/annotated-transformer/

    2024

  24. [32]

    Falcon and The PyTorch Lightning team, PyTorch Lightning

    W. Falcon and The PyTorch Lightning team, PyTorch Lightning. (Mar. 2019). doi: 10.5281/zenodo.3828935

    2019 doi

  25. [33]

    Self-Supervised Speech Representation Learning: A Review,

    A. Mohamed et al., “Self-Supervised Speech Representation Learning: A Review,” IEEE J. Sel. Top. Signal Process., vol. 16, no. 6, pp. 1179–1210, Oct. 2022, doi: 10.1109/JSTSP.2022.3207050

    2022

  26. [34]

    Self-Attention with Relative Position Representations,

    P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-Attention with Relative Position Representations,” Apr. 12, 2018, arXiv: arXiv:1803.02155. doi: 10.48550/arXiv.1803.02155

    2018 doi arXiv

  27. [35]

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,

    C. Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” Sep. 19, 2023, arXiv: arXiv:1910.10683. doi: 10.48550/arXiv.1910.10683. Supplemental Figure 1: Extendable attention mechanism implementation of AttentionSmithy. While no alte...

    2023 doi arXiv

  28. [2025]

    Available: https://github.com/tensorflow/models/tree/master/official/nlp/modeling

    [Online]. Available: https://github.com/tensorflow/models/tree/master/official/nlp/modeling

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.