Pith. sign in

REVIEW 3 major objections 4 minor 27 references

Encoders Help You Disambiguate Word Senses in Neural Machine Translation

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Encoder hidden states, not word embeddings or decoders, carry most of the information needed to disambiguate word senses in neural machine translation.

desk verdict Useful large-scale probing evidence for encoder-side WSD information, but the causal framing outstrips the probe and the instance-level split likely leaks sentence identity. read the letter →

arxiv 1908.11771 v2 pith:FL62C774 submitted 2019-08-30 cs.CL

Gongbo Tang , Rico Sennrich , Joakim Nivre This is my paper · ORCID
classification cs.CL
keywords wordsensedisambiguationneuralmachinetranslationencoderhiddenstatesprobingclassifierself-attentionattentionentropyTransformerRNN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Neural machine translation systems translate ambiguous words better than older methods, but where the disambiguation actually happens inside the network has been unclear. This paper trains a small classifier that, given the encoder's hidden-state representation of an ambiguous noun and a translation candidate, predicts whether the translation matches the correct sense. Across two language pairs and two architectures (RNN and Transformer), the classifier jumps from roughly 63–69 percent accuracy on word embeddings to over 91 percent on encoder hidden states, and accuracy rises consistently with encoder depth. Decoder hidden states do slightly better still, but the leap from embeddings to encoders is so large that the authors conclude encoders do most of the disambiguation work. A separate analysis of self-attention weights and entropy shows the network spends more attention on context words for ambiguous nouns than for nouns in general.

What carries the argument

The central object is the probing classifier: a one-hidden-layer feed-forward network trained to decide whether a translation candidate is the correct sense for an ambiguous noun, fed the concatenation of the noun's representation (word embedding, encoder hidden state, or decoder hidden state) and the candidate's word embedding. The layer-dependent hidden states and the self-attention distributions of Transformer encoders are the other two load-bearing parts. The classifier converts 'information relevant for disambiguation' into a measurable accuracy number, and attention entropy (the Shannon entropy of a token's attention distribution, averaged over heads) turns attention concentration into a comparable statistic. Together they let the authors compare representations across layers, architectures, and the encoder–decoder boundary.

What would settle it

Replace the encoder hidden state of an ambiguous noun with its context-free word embedding at inference time and measure the drop in translation accuracy for that noun; if the translation is unaffected, the hidden state is not what carries the disambiguation signal.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that encoder hidden states in NMT contain rich, layer-dependent information about the correct sense of ambiguous source nouns, and that this information is already largely present in the first encoder layer. Training a feed-forward classifier on the concatenation of an ambiguous noun's hidden state and a translation candidate's embedding yields 91–97 percent accuracy on the ContraWSD-derived test instances, compared with 63–69 percent when the same classifier sees only static word embeddings. Accuracy increases monotonically with encoder depth for Transformers and with each bidirectional RNN layer, so higher layers encode more sense-relevant context. Decoder hidden states, extracted under constrained decoding, push accuracy a few points higher, but the embedding-to-encoder gain is roughly ten times larger, which the authors take as evidence that 'most of the disambiguation work is done by encoders.' The self-attention analysis complements this: in the first layer attention concentrates on the ambiguous noun itself; in later layers it shifts to context words, and ambiguous nouns receive less self-weight and lower attention entropy than nouns in general, indicating focused context gathering.

Load-bearing premise

The load-bearing premise is that a classifier's ability to read sense information out of hidden states means the NMT model itself uses that information to choose its translations; the paper measures the first, not the second.

Editorial extensions

If this is right

  • Higher encoder layers can be used as sense-aware source representations for downstream tasks, since sense information accumulates with depth.
  • The first encoder layer already provides most of the disambiguation signal, so layer-wise analyses of NMT should not assume that sense information appears only near the output.
  • The probe method transfers across language pairs and architectures, offering a reusable diagnostic for where disambiguation happens in any NMT model.
  • Self-attention's shift from self-focused to context-focused distributions across layers gives an interpretable signature of ambiguity handling in Transformers.
  • Since decoder states add only a few points over encoder states, future work on sense-aware decoding should still expect encoders to supply the bulk of the context.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If encoders truly carry sense information in a layer-wise fashion, then freezing or pruning higher encoder layers should degrade WSD more than lower layers; this is a testable layer-wise ablation the paper does not run.
  • The probe result predicts that an NMT system's translation of ambiguous nouns should be more accurate when the source-side context is preserved in the encoder; a causal test would swap the encoder hidden state of an ambiguous noun for its context-free word embedding at inference and measure the drop in sense accuracy.
  • The attention asymmetry (more context attention for ambiguous nouns) could be used as an unsupervised signal to detect ambiguous words in a source sentence without a labeled WSD test set.
  • The finding that forward RNNs outperform backward RNNs suggests source-side left context matters more than right context for the tested languages; extending the probe to head-final languages or longer-distance ambiguities would show whether this is language-specific or general.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper investigates which component of NMT models, encoder or decoder, is responsible for word-sense disambiguation (WSD). Using instances extracted from the ContraWSD benchmark, the authors train a feed-forward classifier to predict whether a translation candidate for an ambiguous noun is correct, using as input either the static embedding of the noun, the encoder hidden state at the noun's position, or the decoder hidden state at the step generating that translation. Experiments on German→English and German→French with both RNN and Transformer NMT models show that encoder hidden states improve accuracy over embeddings by 23 to 34 percentage points, that decoder states improve further by a few points, and that accuracy rises with encoder depth. A complementary analysis of self-attention weights and entropy suggests that the Transformer encoder attends more to context for ambiguous nouns and that the first layer focuses on the noun itself while higher layers use context. The paper concludes that encoders encode substantial WSD-relevant information and that 'most of the disambiguation work is done by encoders.'

Significance. If the representational claim holds, this is a valuable large-scale demonstration that NMT encoder hidden states carry context-dependent WSD information, contradicting the earlier conclusion of Marvin and Koehn (2018) and extending probing analysis to two language pairs and two architectures. The experimental protocol is solid in several respects: the evaluation uses an external benchmark (ContraWSD) as ground truth, results are averaged over 10 classifier seeds, and the accuracy gaps are large. The attention analysis is also a useful step toward understanding how self-attention might support disambiguation. However, the paper's central interpretive claims go beyond what the probe can establish, and the causal framing—'most of the disambiguation work is done by encoders'—is not supported by the experiments as reported.

major comments (3)
  1. [§2.1, Table 2] The main evidence that encoder hidden states encode WSD-relevant information, the 23–34 point gap between ENC and Embedding in Table 2, does not rule out the possibility that the classifier is exploiting a more decodable encoding of the ambiguous noun's identity rather than sentential context. Since the classifier input concatenates the ambiguous noun's representation with the candidate embedding, and the baseline already provides the noun's identity through its word embedding, a control that pairs the ambiguous noun's hidden state with a mismatched or shuffled context (e.g., the same noun from a different sentence) is necessary to attribute the accuracy to context-dependent disambiguation. Without such a control, the first finding—'the higher the encoder layer, the more relevant information is encoded'—remains underdetermined, because higher layers could also be encoding more about the noun's surface form.
  2. [§4.1.3] The conclusion that 'most of the disambiguation work is done by encoders' is a causal claim that the probe design cannot support. The decoder hidden state in the DEC condition is taken from the time step that actually generates the translation of the ambiguous noun under constrained decoding; this hidden state is the very representation used to predict that target word, so the classifier has direct access to a cue (the word prediction signal) that is absent in the ENC condition. The comparisons among Embedding, ENC, and DEC in Table 2 therefore conflate representational content with causal contribution. To support the attribution of work, the paper would need an intervention such as an encoder-free NMT model (e.g., the simplification studied in Tang et al. 2019b) or an analysis that ablates encoder information while controlling the decoder input.
  3. [§4.2.1–4.2.2] The attention analysis compares ambiguous nouns to all nouns without controlling for potentially confounding factors such as token frequency, word length, or sentence position. Since ambiguous nouns may be rarer or longer than typical nouns, the observed lower self-attention weight and lower entropy could reflect these lexical properties rather than the model's recognition of ambiguity. Matching ambiguous and non-ambiguous nouns on these properties, or including them as covariates in a regression, would make the 'self-attention can detect ambiguous nouns' finding more robust.
minor comments (4)
  1. [Abstract] The note 'this is a revised version. The content related to decoder hidden states has been updated' is not appropriate in an abstract and should be moved to a footnote or removed entirely.
  2. [§2.1] In the paragraph on decoders, 'To get these hidden states. we force NMT models' is a sentence fragment with a typo; it should read 'To get these hidden states, we force NMT models...'.
  3. [References] The author name 'Voita' appears as 'V oita' with a space in several references; this should be corrected.
  4. [Figures 3 and 4] The attention weight and entropy plots show mean differences without error bars or significance tests; adding these would help the reader judge whether the reported differences are reliable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: the probing results are measured against an external benchmark, and the interpretive leap from probe accuracy to mechanism is overstatement, not circular reduction.

full rationale

The paper's derivation chain is not circular. The central measurements are supervised probing accuracies of independently trained NMT models on a held-out split of the external ContraWSD benchmark. The classifier input ('the concatenation of Rambi and Rsense') and the binary labels ('whether it corresponds to the correct sense') are defined in Section 2.1, and the accuracy values in Table 2 (e.g., 'ENC achieves much higher accuracy than Embedding... 94.2 97.2 91.7 95.6') are empirical outputs, not consequences of any equation fitted to the paper's conclusions. No parameter is fitted to the target claim that encoders encode sense information; the probe is trained only to predict translation-candidate correctness. The later statement that 'most of the disambiguation work is done by encoders' is an interpretive generalization from the observed accuracy gap, and the paper does not directly test whether the NMT system actually uses the probed information, but such overinterpretation is a correctness risk, not a circular reduction. Author overlap on the ContraWSD resource (Rios et al., 2017) and the prior speculation by Tang et al. (2018b) are cited as background or test data, but the benchmark supplies external ground truth, the results are reproduced across two architectures and two language pairs, and no load-bearing step reduces to a self-citation or to a fitted input renamed as a prediction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical quantities; all quantities are standard ML representations. The main assumptions are interpretive, not formal axioms, and they are explicit about the intended link between classifier accuracy and model behavior.

assumptions (3)
  • domain assumption High accuracy of a supervised probing classifier on hidden states indicates the encoder has encoded relevant information for WSD.
    Invoked in Sections 2.1 and 3.1 to move from classifier accuracy to claims about the encoder's representational content.
  • domain assumption Attention weights reflect the degree of contribution to the current word representation.
    Stated at the start of Section 2.2: 'The attention weights can be viewed as the degree of contribution to the current word representation', which underlies the interpretation of self-attention patterns.
  • domain assumption ContraWSD provides correct sense labels for the ambiguous nouns.
    Used throughout as ground truth for training and evaluating the WSD classifier; the paper does not audit ContraWSD labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Encoders Help You Disambiguate Word Senses in Neural Machine Translation." pith.science (2026). https://pith.science/paper/FL62C774

@misc{pith2026190811771,
  author       = {Pith},
  title        = {Pith review of: Encoders Help You Disambiguate Word Senses in Neural Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FL62C774}},
  note         = {Machine review of arXiv:1908.11771}
}
read the original abstract

Neural machine translation (NMT) has achieved new state-of-the-art performance in translating ambiguous words. However, it is still unclear which component dominates the process of disambiguation. In this paper, we explore the ability of NMT encoders and decoders to disambiguate word senses by evaluating hidden states and investigating the distributions of self-attention. We train a classifier to predict whether a translation is correct given the representation of an ambiguous noun. We find that encoder hidden states outperform word embeddings significantly which indicates that encoders adequately encode relevant information for disambiguation into hidden states. Decoders could provide further relevant information for disambiguation. Moreover, the attention weights and attention entropy show that self-attention can detect ambiguous nouns and distribute more attention to the context. Note that this is a revised version. The content related to decoder hidden states has been updated.

Figures

Figures reproduced from arXiv: 1908.11771 by the authors.

Figure 1
Figure 1. Illustration of the WSD classification task, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. illustrates WSD accuracy in different en￾coder layers, with standard deviation as error bars. Even the hidden states from the first layer boost the WSD performance substantially compared to using word embeddings. This means that most of the relevant information for WSD has been en￾coded into hidden states in the first encoder layer. For Transformers, the WSD accuracy goes up con￾sistently as the encoder layer gets h… view at source ↗
Figure 4
Figure 4. The average attention entropy of ambiguous [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 22 canonical work pages

  1. [1]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. https://arxiv.org/abs/1409.0473 Neural machine translation by jointly learning to align and translate . In Proceedings of the 3rd International Conference on Learning Representations, San Diego, California, USA

    arXiv 2015

  2. [2]

    Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017 a . http://aclweb.org/anthology/P17-1080 What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861--872, Vancouver, Canada. Association for ...

    work page 2017

  3. [3]

    Yonatan Belinkov, Llu\' i s M\` a rquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017 b . http://www.aclweb.org/anthology/I17-1001 Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (V...

    work page 2017

  4. [4]

    Ond r ej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. http://aclweb.org/anthology/W17-4717 Findings of the 2017 conference on machine translation ( WMT 17) . In...

    work page 2017

  5. [5]

    Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. http://www.aclweb.org/anthology/D14-1179 Learning phrase representations using RNN encoder--decoder for statistical machine translation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing,...

    work page 2014

  6. [6]

    Hamidreza Ghader and Christof Monz. 2017. http://www.aclweb.org/anthology/I17-1004 What does attention in neural machine translation pay attention to? In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 30--39, Taipei, Taiwan. Asian Federation of Natural Language Processing

    work page 2017

  7. [7]

    Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2017. http://arxiv.org/abs/1712.05690 Sockeye: A toolkit for neural machine translation . arXiv preprint arXiv:1712.05690

    arXiv 2017

  8. [8]

    Nal Kalchbrenner and Phil Blunsom. 2013. http://www.aclweb.org/anthology/D13-1176 Recurrent continuous translation models . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700--1709, Seattle, Washington, USA. Association for Computational Linguistics

    work page 2013

Show all 27 references
  1. [9]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. https://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In Proceedings of the 3rd International Conference on Learning Representations, San Diego, California, USA

    2015 arXiv

  2. [10]

    Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation . In Proceedings of the 10th Machine Translation Summit , pages 79--86, Phuket, Thailand

    2005

  3. [11]

    Philipp Koehn and Rebecca Knowles. 2017. http://www.aclweb.org/anthology/W17-3204 Six challenges for neural machine translation . In Proceedings of the First Workshop on Neural Machine Translation, pages 28--39, Vancouver, Canada. Association for Computational Linguistics

    2017

  4. [12]

    Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. http://aclweb.org/anthology/D15-1166 Effective approaches to attention-based neural machine translation . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412--1421, Lisb...

    2015

  5. [13]

    Rebecca Marvin and Phillip Koehn. 2018. https://amtaweb.org/wp-content/uploads/2018/03/AMTA_2018_Proceedings_Research_Track.pdf#page=131 Exploring word sense disambiguation abilities of neural machine translation systems . In Proceedings of AMTA 2018 (Volume 1: MT Research Tra...

    2018

  6. [14]

    Matt Post. 2018. http://aclweb.org/anthology/W18-6319 A call for clarity in reporting bleu scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191. Association for Computational Linguistics

    2018

  7. [15]

    Matt Post and David Vilar. 2018. https://www.aclweb.org/anthology/N18-1119 Fast lexically constrained decoding with dynamic beam allocation for neural machine translation . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computationa...

    2018

  8. [16]

    Annette Rios, Laura Mascarell, and Rico Sennrich. 2017. https://www.aclweb.org/anthology/W17-4702 Improving word sense disambiguation in neural machine translation with sense embeddings . In Proceedings of the Second Conference on Machine Translation, pages 11--19, Copenhagen,...

    2017

  9. [17]

    Annette Rios, Mathias M \"u ller, and Rico Sennrich. 2018. https://www.aclweb.org/anthology/W18-6437 The word sense disambiguation test suite at WMT 18 . In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 588--596, Belgium, Brussels. Assoc...

    2018

  10. [18]

    Helmut Schmid. 1995. Improvements in part-of-speech tagging with an application to german. In Proceedings of the ACL SIGDAT-Workshop, Dublin, Ireland. Association for Computational Linguistics

    1995

  11. [19]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. http://www.aclweb.org/anthology/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2016

  12. [20]

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf Sequence to sequence learning with neural networks . In Proceedings of the Neural Information Processing Systems 2014, pages 3104--3112,...

    2014

  13. [21]

    Gongbo Tang, Mathias M \"u ller, Annette Rios, and Rico Sennrich. 2018 a . http://aclweb.org/anthology/D18-1458 Why self-attention? a targeted evaluation of neural machine translation architectures . In Proceedings of the 2018 Conference on Empirical Methods in Natural Languag...

    2018

  14. [22]

    Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2018 b . http://aclweb.org/anthology/W18-6304 An analysis of attention mechanisms: The case of word sense disambiguation in neural machine translation . In Proceedings of the Third Conference on Machine Translation: Research Papers...

    2018

  15. [23]

    Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2019 a . https://doi.org/10.18653/v1/D19-1149 Encoders help you disambiguate word senses in neural machine translation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internat...

    2019 doi

  16. [24]

    Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2019 b . Understanding neural machine translation by simplification: The case of encoder-free models. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2019 , Varna, Bulgaria

    2019

  17. [25]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention is all you need . In Advances in Neural Information Processing Systems 30,...

    2017

  18. [26]

    Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. https://www.aclweb.org/anthology/P18-1117 Context-aware neural machine translation learns anaphora resolution . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume...

    2018

  19. [27]

    Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. https://www.aclweb.org/anthology/P19-1580 Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned . In Proceedings of the 57th Annual Meeting of the Assoc...

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.