{"id":"dbb9b6ec-8f5c-45f1-9d62-cca62a522933","arxiv_id":"1908.11771","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Encoder hidden states in NMT encode far more word-sense disambiguation information than word embeddings, with higher layers and self-attention focusing on context contributing most.","lead":"The paper probes neural machine translation models and finds that encoder hidden states carry far more word-sense information than static word embeddings. It also finds self-attention focuses on ambiguous words first, then shifts to context, while decoders only add a small extra disambiguation gain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probing accuracy cannot support the causal claim that encoders do the disambiguation work; the decisive missing control is whether the classifier's input representation carries the source word's identity plus translation-candidate info, which would let it trivially separate senses without testing…","rationale":"I agree with the reader's weakest assumption: probing demonstrates representational sufficiency but not causal use by the NMT system. The most load-bearing unaddressed issue is not the causality per se but the possibility that the probe's high accuracy does not even establish context-sensitive representation, because the input includes the ambiguous noun's own representation and the translation candidate's embedding. The paper compares ENC against a word-embedding baseline, but that baseline uses the same candidate embedding and the same noun embedding; the fact that ENC beats it by 23–34 points strongly suggests sentential context is useful, yet no control isolates whether the classifier is learning the mapping from specific source nouns to correct candidate senses (a sense inventory lookup that requires no context). Because the ContraWSD instances are generated with a small candidate set, a classifier could memorize source-noun-to-sense patterns and still earn high accuracy on the held-out test if the test nouns are the same lexical types with the same sense distributions. The layer-wise result compounds the ambiguity: higher layers may encode more about the source sentence's lexical content, not about disambiguating context. This concern is concrete and testable with a shuffling or noun-embedding baseline, so the verdict remains CONDITIONAL rather than REJECT; the paper's headline empirical finding—hidden states outperform embeddings—would survive the control, but the interpretive claims about layer-wise 'relevant information' and 'most disambiguation work by encoders' would need to be downgraded.","tokens_in":7897,"tokens_out":1635,"duration_ms":14590,"concrete_test":"Train the same probe with a shuffled-context control: feed the hidden state of the ambiguous noun from sentence A paired with the translation candidate from sentence B of the same source noun type (or randomly permute the context tokens when computing the encoder state). If accuracy on correctly paired instances remains near the original ENC accuracy, the probe is using noun identity plus candidate identity rather than sense-bearing context. Alternatively, train a probe on the concatenation of the source noun's embedding and the candidate embedding with no positional or sentential information; if that baseline already approaches the ENC accuracy, then the hidden states add little disambiguation signal beyond word identity. A second check is to ablate the target noun's own token from the Transformer encoder state and recompute probe accuracy.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that encoder hidden states are a rich, layer-dependent source of WSD information and that 'most of the disambiguation work is done by encoders' (Section 4.1.3). The experiments show a supervised probe can predict translation correctness from encoder states, but they do not rule out the alternative that the probe is exploiting shallow cues—most plausibly the identity of the ambiguous source noun plus the target candidate embedding. Since the classifier input concatenates the ambiguous noun representation with the candidate embedding, and the baseline embedding condition already reaches 63% accuracy on a balanced task, the question is whether ENC accuracy reflects disambiguation-relevant context or simply reflects that the representation identifies which source noun is in the sentence. The paper never reports a control where the ambiguous noun and the target candidate are paired with a mismatched or shuffled context. The finding that higher layers improve accuracy is also consistent with the layers storing more about source surface form, not more about sense. Thus the paper's phrasing 'encoded a lot of relevant information for WSD' and 'most of the disambiguation work is done by encoders' overstates what probing can establish; the representational sufficiency claim is strong, the mechanistic claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates which component of NMT models, encoder or decoder, is responsible for word-sense disambiguation (WSD). Using instances extracted from the ContraWSD benchmark, the authors train a feed-forward classifier to predict whether a translation candidate for an ambiguous noun is correct, using as input either the static embedding of the noun, the encoder hidden state at the noun's position, or the decoder hidden state at the step generating that translation. Experiments on German→English and German→French with both RNN and Transformer NMT models show that encoder hidden states improve accuracy over embeddings by 23 to 34 percentage points, that decoder states improve further by a few points, and that accuracy rises with encoder depth. A complementary analysis of self-attention weights and entropy suggests that the Transformer encoder attends more to context for ambiguous nouns and that the first layer focuses on the noun itself while higher layers use context. The paper concludes that encoders encode substantial WSD-relevant information and that 'most of the disambiguation work is done by encoders.'","tokens_in":8128,"tokens_out":6493,"duration_ms":59148,"significance":"If the representational claim holds, this is a valuable large-scale demonstration that NMT encoder hidden states carry context-dependent WSD information, contradicting the earlier conclusion of Marvin and Koehn (2018) and extending probing analysis to two language pairs and two architectures. The experimental protocol is solid in several respects: the evaluation uses an external benchmark (ContraWSD) as ground truth, results are averaged over 10 classifier seeds, and the accuracy gaps are large. The attention analysis is also a useful step toward understanding how self-attention might support disambiguation. However, the paper's central interpretive claims go beyond what the probe can establish, and the causal framing—'most of the disambiguation work is done by encoders'—is not supported by the experiments as reported.","major_comments":[{"comment":"The main evidence that encoder hidden states encode WSD-relevant information, the 23–34 point gap between ENC and Embedding in Table 2, does not rule out the possibility that the classifier is exploiting a more decodable encoding of the ambiguous noun's identity rather than sentential context. Since the classifier input concatenates the ambiguous noun's representation with the candidate embedding, and the baseline already provides the noun's identity through its word embedding, a control that pairs the ambiguous noun's hidden state with a mismatched or shuffled context (e.g., the same noun from a different sentence) is necessary to attribute the accuracy to context-dependent disambiguation. Without such a control, the first finding—'the higher the encoder layer, the more relevant information is encoded'—remains underdetermined, because higher layers could also be encoding more about the noun's surface form.","section":"§2.1, Table 2"},{"comment":"The conclusion that 'most of the disambiguation work is done by encoders' is a causal claim that the probe design cannot support. The decoder hidden state in the DEC condition is taken from the time step that actually generates the translation of the ambiguous noun under constrained decoding; this hidden state is the very representation used to predict that target word, so the classifier has direct access to a cue (the word prediction signal) that is absent in the ENC condition. The comparisons among Embedding, ENC, and DEC in Table 2 therefore conflate representational content with causal contribution. To support the attribution of work, the paper would need an intervention such as an encoder-free NMT model (e.g., the simplification studied in Tang et al. 2019b) or an analysis that ablates encoder information while controlling the decoder input.","section":"§4.1.3"},{"comment":"The attention analysis compares ambiguous nouns to all nouns without controlling for potentially confounding factors such as token frequency, word length, or sentence position. Since ambiguous nouns may be rarer or longer than typical nouns, the observed lower self-attention weight and lower entropy could reflect these lexical properties rather than the model's recognition of ambiguity. Matching ambiguous and non-ambiguous nouns on these properties, or including them as covariates in a regression, would make the 'self-attention can detect ambiguous nouns' finding more robust.","section":"§4.2.1–4.2.2"}],"minor_comments":[{"comment":"The note 'this is a revised version. The content related to decoder hidden states has been updated' is not appropriate in an abstract and should be moved to a footnote or removed entirely.","section":"Abstract"},{"comment":"In the paragraph on decoders, 'To get these hidden states. we force NMT models' is a sentence fragment with a typo; it should read 'To get these hidden states, we force NMT models...'.","section":"§2.1"},{"comment":"The author name 'Voita' appears as 'V oita' with a space in several references; this should be corrected.","section":"References"},{"comment":"The attention weight and entropy plots show mean differences without error bars or significance tests; adding these would help the reader judge whether the reported differences are reliable.","section":"Figures 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The paper has been published at EMNLP 2019 and this arXiv version is a revision; however, as a submission the report stands. The missing context-mismatch control and the overinterpretation of probe results are likely to be raised by reviewers, so the authors should address them head-on, either with new experiments or by substantially tempering the causal claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis is a well-executed large-scale probing study. The authors train a classifier to predict whether a translation candidate for an ambiguous noun is correct, using either word embeddings or encoder hidden states from NMT models. The main finding—encoder states carry far more sense-discriminative information than embeddings, and higher layers carry more than lower ones—is credible and directly contradicts Marvin & Koehn (2018). They average over 10 seeds, use two language pairs (DE->EN, DE->FR) and two architectures (RNN and Transformer), and the accuracy gaps are huge (23–34 points). The attention part is a reasonable bonus: ambiguous nouns receive less self-attention weight and lower attention entropy than nouns in general, suggesting the model focuses on context. Credit is earned here.\n\nThe soft spots are about interpretation and one methodological detail.\n\nThe interpretation: Section 4.1.3 concludes \"most of the disambiguation work is done by encoders.\" But the experiments only show an external probe can recover sense information from encoder states—representational sufficiency, not causal mechanism. The decoder might not actually use that information; the title and conclusions push past what the evidence allows. This is a common probing pitfall, and here it is load-bearing.\n\nThe methodology: The train/test split appears to be instance-level random sampling. The data are derived from about 7k sentences, with roughly 2.4 candidate instances per ambiguous noun. Randomly splitting instances almost guarantees the same source sentence appears in both train and test. Because exactly one candidate is correct per sentence, seeing a different candidate for that sentence in training can tell you whether the test candidate is correct—a form of leakage that inflates accuracy. The paper reports no sentence-level split. This is a real concern, not a nitpick, and it directly affects the headline numbers. The noun-identity control raised elsewhere seems covered by the embedding baseline (63% accuracy with embeddings shows noun+candidate alone is insufficient), but the sentence-overlap issue is not.\n\nNo code or data is released, which is a minor practical annoyance.\n\nBottom line: This paper deserves review, not a desk reject. A serious referee should ask for a sentence-level split or an overlap analysis, and for the causal claims to be softened or supported by a follow-up experiment (e.g., interfering with encoder states). If those numbers hold, the layer-wise finding is a useful data point for anyone working on NMT interpretability.\n\nI'd bring it to a reading group to discuss the probing methodology, and I'd cite it cautiously for the empirical result. Yes to peer review.","headline":"Useful large-scale probing evidence for encoder-side WSD information, but the causal framing outstrips the probe and the instance-level split likely leaks sentence identity.","tokens_in":8634,"tokens_out":6648,"would_cite":true,"duration_ms":62772,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Encoder hidden states, not word embeddings or decoders, carry most of the information needed to disambiguate word senses in neural machine translation.","keywords":["word sense disambiguation","neural machine translation","encoder hidden states","probing classifier","self-attention","attention entropy","Transformer","RNN"],"falsifier":"Replace the encoder hidden state of an ambiguous noun with its context-free word embedding at inference time and measure the drop in translation accuracy for that noun; if the translation is unaffected, the hidden state is not what carries the disambiguation signal.","tokens_in":7710,"feed_emoji":"🧠","tokens_out":5856,"duration_ms":44463,"temperature":0.7,"pith_summary":"Neural machine translation systems translate ambiguous words better than older methods, but where the disambiguation actually happens inside the network has been unclear. This paper trains a small classifier that, given the encoder's hidden-state representation of an ambiguous noun and a translation candidate, predicts whether the translation matches the correct sense. Across two language pairs and two architectures (RNN and Transformer), the classifier jumps from roughly 63–69 percent accuracy on word embeddings to over 91 percent on encoder hidden states, and accuracy rises consistently with encoder depth. Decoder hidden states do slightly better still, but the leap from embeddings to encoders is so large that the authors conclude encoders do most of the disambiguation work. A separate analysis of self-attention weights and entropy shows the network spends more attention on context words for ambiguous nouns than for nouns in general.","feed_headline":"Encoders carry most of the disambiguation work in NMT","feed_subtitle":"A probe classifier jumps from roughly 63% on word embeddings to over 91% on encoder hidden states.","key_machinery":"The central object is the probing classifier: a one-hidden-layer feed-forward network trained to decide whether a translation candidate is the correct sense for an ambiguous noun, fed the concatenation of the noun's representation (word embedding, encoder hidden state, or decoder hidden state) and the candidate's word embedding. The layer-dependent hidden states and the self-attention distributions of Transformer encoders are the other two load-bearing parts. The classifier converts 'information relevant for disambiguation' into a measurable accuracy number, and attention entropy (the Shannon entropy of a token's attention distribution, averaged over heads) turns attention concentration into a comparable statistic. Together they let the authors compare representations across layers, architectures, and the encoder–decoder boundary.","core_discovery":"On the paper's own terms, the discovery is that encoder hidden states in NMT contain rich, layer-dependent information about the correct sense of ambiguous source nouns, and that this information is already largely present in the first encoder layer. Training a feed-forward classifier on the concatenation of an ambiguous noun's hidden state and a translation candidate's embedding yields 91–97 percent accuracy on the ContraWSD-derived test instances, compared with 63–69 percent when the same classifier sees only static word embeddings. Accuracy increases monotonically with encoder depth for Transformers and with each bidirectional RNN layer, so higher layers encode more sense-relevant context. Decoder hidden states, extracted under constrained decoding, push accuracy a few points higher, but the embedding-to-encoder gain is roughly ten times larger, which the authors take as evidence that 'most of the disambiguation work is done by encoders.' The self-attention analysis complements this: in the first layer attention concentrates on the ambiguous noun itself; in later layers it shifts to context words, and ambiguous nouns receive less self-weight and lower attention entropy than nouns in general, indicating focused context gathering.","pith_inferences":["If encoders truly carry sense information in a layer-wise fashion, then freezing or pruning higher encoder layers should degrade WSD more than lower layers; this is a testable layer-wise ablation the paper does not run.","The probe result predicts that an NMT system's translation of ambiguous nouns should be more accurate when the source-side context is preserved in the encoder; a causal test would swap the encoder hidden state of an ambiguous noun for its context-free word embedding at inference and measure the drop in sense accuracy.","The attention asymmetry (more context attention for ambiguous nouns) could be used as an unsupervised signal to detect ambiguous words in a source sentence without a labeled WSD test set.","The finding that forward RNNs outperform backward RNNs suggests source-side left context matters more than right context for the tested languages; extending the probe to head-final languages or longer-distance ambiguities would show whether this is language-specific or general."],"forward_implications":["Higher encoder layers can be used as sense-aware source representations for downstream tasks, since sense information accumulates with depth.","The first encoder layer already provides most of the disambiguation signal, so layer-wise analyses of NMT should not assume that sense information appears only near the output.","The probe method transfers across language pairs and architectures, offering a reusable diagnostic for where disambiguation happens in any NMT model.","Self-attention's shift from self-focused to context-focused distributions across layers gives an interpretable signature of ambiguity handling in Transformers.","Since decoder states add only a few points over encoder states, future work on sense-aware decoding should still expect encoders to supply the bulk of the context."],"supporting_citations":[{"why":"Supplies the ContraWSD test set and the sense-candidate instances the classifier is trained and evaluated on.","marker":"(Rios et al., 2017)"},{"why":"The earlier study claiming higher encoder layers do not improve disambiguation, which this paper revisits with larger data.","marker":"(Marvin and Koehn, 2018)"},{"why":"Shows higher NMT layers encode more morphology, motivating the layer-depth hypothesis.","marker":"(Belinkov et al., 2017a)"},{"why":"Shows higher layers improve on semantic tagging, again motivating depth-based analysis.","marker":"(Belinkov et al., 2017b)"},{"why":"Provides the attention-entropy measure and the observation that nouns receive distinct attention distributions.","marker":"(Ghader and Monz, 2017)"},{"why":"Establishes that Transformer-based NMT outperforms RNN-based NMT on WSD, which the encoder-probe results corroborate.","marker":"(Tang et al., 2018a)"},{"why":"Provides the constrained-decoding method used to extract decoder hidden states for the decoder probe.","marker":"(Post and Vilar, 2018)"}],"fun_headline_variants":["Encoders do most of the word-sense disambiguation in NMT","NMT ambiguity resolved by encoder hidden states, not embeddings","Probing encoders reveals where NMT sense disambiguation happens","Encoder layers, not word embeddings, carry NMT sense cues","Attention and encoders: the primary disambiguation engine in NMT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a classifier's ability to read sense information out of hidden states means the NMT model itself uses that information to choose its translations; the paper measures the first, not the second.","fun_headline_variants_meta":{"raw":{"variants":["Encoders do most of the word-sense disambiguation in NMT","NMT ambiguity resolved by encoder hidden states, not embeddings","Probing encoders reveals where NMT sense disambiguation happens","Encoder layers, not word embeddings, carry NMT sense cues","Attention and encoders: the primary disambiguation engine in NMT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001369,"raw_usage":{"total_tokens":5532,"prompt_tokens":912,"completion_tokens":4620,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":4532}},"tokens_in":528,"tokens_out":4620,"duration_ms":28959,"temperature":1.0,"reasoning_tokens":4532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:06:32.377897+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the encoder hidden state of an ambiguous noun with its context-free word embedding at inference time and measure the drop in translation accuracy for that noun; if the translation is unaffected, the hidden state is not what carries the disambiguation signal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ContraWSD test set and the sense-candidate instances the classifier is trained and evaluated on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The earlier study claiming higher encoder layers do not improve disambiguation, which this paper revisits with larger data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the attention-entropy measure and the observation that nouns receive distinct attention distributions."}],"review_version":1}