Pith. sign in

REVIEW 5 major objections 5 minor 5 cited by

Self-Knowledge Distillation in Natural Language Processing

T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A model's own word distances lift BLEU and lower LM loss.

desk verdict A genuinely new teacher-free distillation idea with thin empirical support; the Euclidean-distance proxy for soft targets is unvalidated, and the paper needs more than a good idea to be published. read the letter →

arxiv 1908.01851 v1 pith:O5CJU4SZ submitted 2019-08-02 cs.CL cs.LGstat.ML

Sangchul Hahn , Heeyoul Choi This is my paper · ORCID
classification cs.CLcs.LGstat.ML
keywords self-knowledgedistillationknowledgewordembeddingslanguagemodelingneuralmachinetranslationsofttargetprobabilitiescross-entropyBLEU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a neural text-generation model can be trained better by distilling knowledge from itself: the model's own word-embedding space, right below the softmax layer, is used to build a soft target distribution during training. The proposed method, self-knowledge distillation (SKD), takes the predicted word and the target word and assigns probability mass to the predicted word based on how close its embedding is to the target embedding. On language modeling (Penn TreeBank, Wiki-2) this lowers test negative log-likelihood below both the baseline and a Gaussian-noise-injected baseline, and on neural machine translation (WMT En-Fi, Fi-En, En-De) it improves BLEU by roughly one point. The paper also reports that SKD slows overfitting. If the claim is right, any cross-entropy-trained generation model can be improved by a simple change to its objective, without a pretrained teacher.

What carries the argument

The central object is the approximate soft target probability built from Euclidean distance in the embedding space. For each training step, SKD replaces the full softmax distribution with a two-point distribution: $q_n = \min\{\exp(-\sigma\|w_t - w_n\|^2), 0.5\}$, $q_t = 1 - q_n$, where $w_t$ and $w_n$ are the target and predicted word embeddings. This distance is used as a cheap proxy for what the model's softmax would assign to the predicted word, avoiding the cost of computing probabilities over the full vocabulary. The objective $J(\theta) = -(1 - \alpha q_n) \log p_t - \alpha q_n \log p_n$ then mixes the standard cross-entropy with a self-distillation term, with $\alpha$ starting at 0 and increasing to 1 so the model first learns ordinary predictions and only later leans on its own embedding geometry.

What would settle it

Train the same language model or translation model with the SKD objective but compute $q_n$ from the model's actual softmax probabilities for the predicted class instead of from embedding distance, or from distances between randomly permuted word embeddings. If the gain over baseline disappears or becomes equal to the gain from random distances, then the improvement is not caused by geometric knowledge in the embedding space, and the claim that SKD distills word-embedding knowledge would be unsupported.

Watch

Extended reading notes

Core claim

The central claim is that the word-embedding layer below the softmax contains knowledge about word relationships that the one-hot cross-entropy objective throws away, and that a model can extract that knowledge from itself while it trains. Concretely, when the model predicts word $n$ for target $t$, SKD estimates a two-point soft target $q_n = \min\{\exp(-\sigma\|w_t - w_n\|^2), 0.5\}$ and $q_t = 1 - q_n$, and trains with $J(\theta) = -(1 - \alpha q_n) \log p_t - \alpha q_n \log p_n$, where $\alpha$ is ramped from 0 to 1 over training. The paper reports that this objective improves language-model negative log-likelihood (101.40 to 99.38 on PTB, 119.49 to 116.85 on Wiki-2) and translation BLEU (for example, 9.01 to 9.87 for En-Fi at beam width 12), with further gains when combined with Gaussian noise on the decoder's target-word embedding. The authors interpret these results as the model using similarity information in its own embedding geometry to train better than one-hot targets alone.

Load-bearing premise

The method stands on the assumption that the Euclidean distance between the target word's embedding and the predicted word's embedding is a reliable measure of how much probability the model's softmax would assign to the predicted word; if that proxy is wrong, the auxiliary term is just an arbitrary regularizer.

Editorial extensions

If this is right

  • SKD changes only the objective function, so it can be dropped into any word-level generation task trained with cross-entropy, without architectural changes or a pretrained teacher.
  • Language modeling should see lower test negative log-likelihood on word-level corpora; the paper demonstrates this on Penn TreeBank and Wiki-2.
  • Neural machine translation should see roughly a one-point BLEU gain on directions such as En-Fi, Fi-En, and En-De, with qualitative examples suggesting output closer to the reference in phrasing.
  • Models trained with SKD should overfit more slowly than a cross-entropy baseline, since the soft target carries information about the input beyond the hard label.
  • SKD and embedding noise injection are complementary; combining them gives larger gains than either alone, indicating the two supply different kinds of training signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the embedding-distance proxy is valid, the method should generalize beyond two points: extending $q_n$ to the top-$k$ nearest neighbours in the embedding space would give a smoother approximate soft target and likely stronger regularization; the paper does not test this.
  • The same recipe could transfer to any softmax classifier whose pre-softmax representation has meaningful geometry, such as image classification with feature vectors, provided confusable classes sit close in feature space; the paper notes this as future work but does not claim it.
  • The ramp on $\alpha$ implies the method depends on the model's embedding geometry becoming informative early enough; a testable corollary is that gains should shrink with random or poorly initialized embeddings and grow with pretrained embeddings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript proposes Self-Knowledge Distillation (SKD), a training objective for NLP sequence models that derives soft targets from the current model's own word embedding space rather than from a pretrained teacher. Specifically, Eq. (7) defines a soft distribution q_k over vocabulary words from Euclidean distance to the target embedding; Eq. (8) retains only the target class and the model's predicted class, with q_n clipped at 0.5; and Eq. (12) combines the usual cross-entropy with an auxiliary cross-entropy term weighted by a scheduled coefficient α. The method is evaluated on language modeling (PTB, Wiki-2) with NLL and on three WMT'15 translation directions with BLEU, reporting improvements over a baseline and over Gaussian noise injection, plus reduced overfitting.

Significance. If the reported results are robust, SKD is an interesting addition to the self-distillation literature because it requires no pretrained teacher and the auxiliary term is cheap to compute. The paper has the merit of explicitly comparing against Gaussian noise injection as a control, and the qualitative translation examples add some plausibility. However, the paper's own load-bearing assumption—that Euclidean distances in embedding space approximate softmax probabilities—is never tested; the method's free parameters (σ, η, K, clipping) are set manually; and several BLEU results are internally inconsistent with the claim that SKD is more informative than noise. The empirical contribution is therefore preliminary, and the interpretation of the auxiliary term as 'distilled knowledge' rather than a geometry-based regularizer remains unsupported without additional experiments.

major comments (5)
  1. [Section 3.1, Eqs. (7)-(8) and Eq. (12)] The load-bearing assumption that q_n from Eq. (8) approximates the model's actual softmax probability p_n is never validated. The paper states this assumption in words but reports no correlation, rank agreement, or qualitative comparison between exp(-σ||w_t - w_n||²) and p_n on real training examples. Because the auxiliary term in Eq. (12) is entirely defined by this proxy, the 'distilled knowledge' interpretation collapses if the proxy is poor; the term would then be an arbitrary geometry-based regularizer. I request a direct empirical check of the proxy during training, including how the chosen σ affects the agreement.
  2. [Sections 4.2 and 4.3, Tables 2 and 3] The method introduces several free parameters—σ, η, K, and the 0.5 clipping threshold in Eq. (8)—that are set manually per dataset with no sensitivity analysis or selection rule. The reported gains are on the order of 1 BLEU or a few NLL points, which is well within the range that manual tuning of such parameters could produce. The manuscript also provides no error bars, multiple seeds, or statistical significance tests, so it is unclear whether the improvements are reliable. At minimum, report a sensitivity sweep over σ and η, and standard deviations across at least three training runs.
  3. [Table 3, Fi-En and En-De rows] Several reported results contradict the paper's claim (Section 4.2) that SKD provides more knowledgeable information than Gaussian noise. In Table 3, Fi-En beam=1 gives Baseline 10.42, +Noise 10.74, and +SKD 10.70; En-De beam=1 gives +Noise 20.69 and +SKD 20.29. These cases show SKD performing worse than noise injection. The abstract's claim that 'our proposed method improves performance' is therefore not uniformly supported by the table, and the inconsistent rows need an explanation or a qualified statement of the claim.
  4. [Section 4.3, Figure 4] The overfitting-reduction claim is not substantiated quantitatively. Figure 4 shows one validation BLEU curve for En-Fi, but the manuscript reports no training-loss curves, no train-validation gap, no early-stopping comparison, and no repeated runs. The claim that 'SKD reduces overfitting problems' appears in the Introduction and Conclusion, but the evidence is anecdotal; please provide numerical train/validation gaps or a controlled comparison with the same training schedule.
  5. [Section 4, general experimental design] The paper does not compare SKD against label smoothing or against conventional teacher-based knowledge distillation. Label smoothing is the natural baseline for a softened two-class target of the type in Eq. (12), and a Hinton-style distillation from a pretrained teacher directly tests whether the gains are specific to 'distilled knowledge' rather than to any additional smoothing. Without these baselines, the paper cannot distinguish the proposed mechanism from a generic regularizer, which is essential given the unvalidated proxy in Eq. (7).
minor comments (5)
  1. [Throughout] The dataset name is spelled inconsistently as 'WiKi-2' in Section 4.2 and 'Wiki-2' in Table 2; please unify the spelling.
  2. [Section 4.2] The evaluation metric is described as 'negative log-likelihood (NLL) for each sentence'; it would be clearer to state explicitly whether the reported numbers are per token, per sentence, or averaged over the test set.
  3. [Figure 3 caption] The phrase 'the qn value become more close to the target' is grammatically unclear and should be rewritten, for example as 'the qn value approaches its clipped maximum as training proceeds.'
  4. [Section 4.3] The text says 'the dictionary size is 10K' for En-Fi and Fi-En and '30K' for En-De; 'vocabulary size' is the more standard term in NMT and would avoid ambiguity with dictionary lookup.
  5. [Section 4.3] The qualitative translation examples are useful, but it is not stated how they were selected; a sentence about random selection or representativeness would strengthen the claim that they illustrate typical behavior.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: SKD's self-referential targets are empirically benchmarked, and the Euclidean proxy is an assumption, not a derived equivalence.

full rationale

The proposed SKD objective (Eq. 12) is deliberately self-referential: the soft target q_n is computed from the training model's own word embeddings (Eqs. 7-8), so there is no separate teacher. That self-reference is part of the method's definition, not a hidden reduction of the paper's empirical claims. The paper's central claims—that SKD lowers NLL on PTB and Wiki-2 and raises BLEU on En-Fi, Fi-En, and En-De—are evaluated on external benchmark test sets (Tables 2-3), so the improvements are not true by construction. The hyperparameters sigma, eta, and K are set before evaluation rather than fitted to the test data, and the paper does not rename a fitted parameter as a prediction. The Euclidean-distance proxy in Eq. (7) is an unvalidated modeling assumption, and the paper itself notes that 'other approaches like inner product would be possible'; the proxy's correctness is a robustness concern, not a circularity. The only self-citations (Ahn et al. 2016; Choi et al. 2017) appear in background statements about embedding transfer and are not load-bearing for the SKD derivation or the experimental claims. No equation in the paper equates the claimed result to its inputs by construction, so there is no specific reduction to report.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method relies on three unevaluated modeling choices: a Gaussian mapping from embedding distance to probability (Eq. 7), a two-class truncation with a 0.5 cap (Eq. 8), and a hand-scheduled ramp controlled by sigma, eta, and K. None of these is derived or ablated, so they are the hidden cost of the method.

free parameters (4)
  • sigma (scale parameter for q_n) = 0.1 (PTB, Wiki-2); not stated for NMT
    Eq. 7 controls how quickly q_n decays with embedding distance; authors say the value depends on average nearest-neighbor distance but give no fitting procedure or sensitivity analysis.
  • eta (alpha increment) = 0.0002 (PTB), 0.00011 (Wiki-2); not stated for NMT
    Eq. 12 and Algorithm 1: controls the ramp speed of the distillation term; chosen per dataset without justification.
  • K (number of cross-entropy-only batches) = 500 batches (PTB), 900 batches (Wiki-2); not stated for NMT
    Algorithm 1: the model is trained without SKD for K batches before distillation starts; value chosen by hand.
  • q_n clipping threshold = 0.5
    Eq. 8: the predicted class probability is capped at 0.5 so it cannot exceed the target; arbitrary and no ablation is provided.
assumptions (4)
  • domain assumption Euclidean distance between word embeddings is inversely related to softmax probability closeness.
    Section 3.1, Eq. 7: q_k = exp(-sigma ||w_t-w_k||^2)/Z assumes a Gaussian form for soft target probabilities from embedding geometry.
  • ad hoc to paper Ignoring all non-target, non-predicted classes in the approximate soft target is acceptable.
    Section 3.1, Eq. 8: only q_t and q_n are kept, with q_t+q_n=1; all other classes' probability mass is discarded.
  • domain assumption A partially trained current model contains useful knowledge after K batches.
    Algorithm 1 starts with cross-entropy for K batches and only then distills from the current model; no criterion is given for when knowledge is usable.
  • ad hoc to paper The auxiliary soft-target term acts as knowledge rather than arbitrary regularization.
    The paper interprets q_n log p_n as distillation of semantic information; this interpretation is assumed, not derived or tested against simpler regularizers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Knowledge Distillation in Natural Language Processing." pith.science (2026). https://pith.science/paper/O5CJU4SZ

@misc{pith2026190801851,
  author       = {Pith},
  title        = {Pith review of: Self-Knowledge Distillation in Natural Language Processing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O5CJU4SZ}},
  note         = {Machine review of arXiv:1908.01851}
}
read the original abstract

Since deep learning became a key player in natural language processing (NLP), many deep learning models have been showing remarkable performances in a variety of NLP tasks, and in some cases, they are even outperforming humans. Such high performance can be explained by efficient knowledge representation of deep learning models. While many methods have been proposed to learn more efficient representation, knowledge distillation from pretrained deep networks suggest that we can use more information from the soft target probability to train other neural networks. In this paper, we propose a new knowledge distillation method self-knowledge distillation, based on the soft target probabilities of the training model itself, where multimode information is distilled from the word embedding space right below the softmax layer. Due to the time complexity, our method approximates the soft target probabilities. In experiments, we applied the proposed method to two different and fundamental NLP tasks: language model and neural machine translation. The experiment results show that our proposed method improves performance on the tasks.

Figures

Figures reproduced from arXiv: 1908.01851 by the authors.

Figure 1
Figure 1. Network architectures of LM and NMT. Word embedding is presented as gray boxes in the models. 3.1 SKD Equations In order to apply knowledge distillation on a cur￾rent training model, we need to obtain soft target probabilities as qk in Eq. (5) for all classes, but they are not available explicitly. However, when the model is trained enough, then the word embed￾ding has such information implicitly. If a word wi is cl… view at source ↗
Figure 2
Figure 2. Given a target class t, a soft target prob￾abilities are obtained based on the distance in the word embedding space. However, only the target class and the predicted class have soft target proba￾bilities in SKD. transits to Eq. (10). To implement the transition, another parameter α is introduced to Eq. (10), lead￾ing to the final objective function as follows. J(θ) = −(1 − αλqn) log pt − αλqn log pn, (11) α starts f… view at source ↗
Figure 4
Figure 4. BLEU scores of validation data while training on En-Fi corpus with four different mod￾els: Baseline, +Noise, +SKD, and +Noise+SKD. The vertical axis indicates BLEU score and the horizontal axis the number of training iteration. 5 Conclusion We proposed a new knowledge distillation method, self-knowledge distillation, from the probabilities of the currently training model itself. The method uses only two soft target … view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: (a) Change of qn value during NMT model training for En-Fi translation task, and (b) scheduling of α value in Eq. (12) of NMT training for En-Fi translation task. (a) shows that when the model is trained more, the qn value become more close to the target. improves furt…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Active Data Curation Effectively Distills Large-Scale Multimodal Models

    cs.CV 2024-11 conditional novelty 7.0 of 10

    Selecting training data by a reference model's loss acts as an implicit distillation, and combining it with explicit distillation yields more FLOP-efficient vision-language models that beat prior SoTA on 27 benchmarks.

  2. Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Dynamic contrastive knowledge distillation with EMA-generated negatives and VQGAN codebook distribution alignment improves compact image restoration students.

  3. From Collapse to Stability: A Knowledge-Driven Ensemble Framework for Scaling Up Click-Through Rate Prediction Models

    cs.IR 2024-11 conditional novelty 6.0 of 10

    KDEF combines knowledge distillation and deep mutual learning with adaptive exam-score weighting so that CTR ensembles with up to ten sub-networks improve instead of collapse.

  4. Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models

    cs.CL 2024-11 conditional novelty 5.0 of 10

    DynSDPB fine-tunes small language models by self-distilling soft labels from the previous mini-batch, with dynamic per-sample temperature and loss weighting.

  5. Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning

    cs.SD 2025-01 conditional novelty 4.0 of 10

    A self-distillation training scheme for audio-visual embeddings progressively replaces labeled triplets with model-generated soft alignments, improving cross-modal retrieval MAP by roughly 2 percent on AVE and VEGAS.

Reference graph

Works this paper leans on

23 extracted references · 16 canonical work pages · cited by 5 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Sungjin Ahn, Heeyoul Choi, Tanel P \" a rnamaa, and Yoshua Bengio. 2016. A neural knowledge language model. CoRR\/ abs/1608.00318:1--10

    arXiv 2016

  4. [4]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate . In Proc. Int'l Conf. on Learning Representations (ICLR)\/

    work page 2015

  5. [5]

    Courville, and Pascal Vincent

    Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell.\/ 35(8):1798--1828

    work page 2013

  6. [6]

    Yoshua Bengio, R \' e jean Ducharme, and Pascal Vincent. 2003. A Neural Probabilistic Language Model . The Journal of Machine Learning Research\/ 3:1137--1155

    work page 2003

  7. [7]

    Heeyoul Choi, Kyunghyun Cho, and Yoshua Bengio. 2017. Context-dependent word representation for neural machine translation. Computer Speech and Language\/ 45:149--160

    work page 2017

  8. [8]

    Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born-again neural networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, July 10-15, 2018\/ . pages 1602--1611

    work page 2018

Show all 23 references
  1. [9]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016\/ . pages 770--778

    2016

  2. [10]

    Hinton, Oriol Vinyals, and Jeffrey Dean

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR\/ abs/1503.02531

    2015 arXiv

  3. [11]

    Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B

    Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Vi \' e gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot trans...

    2017

  4. [12]

    Le, Navdeep Jaitly, and Geoffrey E

    Quoc V. Le, Navdeep Jaitly, and Geoffrey E. Hinton. 2015. A simple way to initialize recurrent networks of rectified linear units. CoRR\/ abs/1504.00941

    2015 arXiv

  5. [13]

    Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The Penn Treebank . Computational Linguistics\/ 19(2):313--330

    1993

  6. [14]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. CoRR\/ abs/1609.07843

    2016 arXiv

  7. [15]

    Tomas Mikolov, Greg Corrado, Kai Chen, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space . In Proc. Int'l Conf. on Learning Representations (ICLR)\/

    2013

  8. [16]

    Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association\/ . pages 1045--1048

    2010

  9. [17]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP)\/ . pages 1532--1543

    2014

  10. [18]

    Joseph Redmon and Ali Farhadi. 2017. YOLO9000: better, faster, stronger. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017\/ . pages 6517--6525

    2017

  11. [19]

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2014. Fitnets: Hints for thin deep nets. CoRR\/ abs/1412.6550

    2014 arXiv

  12. [20]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In 54th Annual Meeting of the Association for Computational Linguistics\/ . pages 1715--1725

    2016

  13. [21]

    Anders S gaard, Yoav Goldberg, and Omer Levy. 2017. A strong baseline for learning cross-lingual word embeddings from sentence alignments. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain...

    2017

  14. [22]

    Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks . In Advances in Neural Information Processing Systems (NIPS)\/

    2014

  15. [23]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 20...

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.