Pith. sign in

REVIEW 3 major objections 5 minor 43 references

Self-Attention with Structural Position Representations

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Augmenting self-attention with dependency-tree depth and distance as position representations yields consistent BLEU gains over sequential positions in Chinese-to-English and English-to-German translation.

desk verdict A modest but real empirical extension of relative position encodings into dependency-tree coordinates, undercut by an undefined fusion function and a relative-position formula that does not compute the tree distance it claims. read the letter →

arxiv 1909.00383 v1 pith:5ANG4BBN submitted 2019-09-01 cs.CL

Xing Wang , Zhaopeng Tu , Longyue Wang , Shuming Shi This is my paper · ORCID
classification cs.CL
keywords self-attentionpositionencodingdependencytreestructuralrepresentationneuralmachinetranslationTransformersyntacticstructure
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the position signal in a self-attention encoder need not be purely sequential: the dependency tree of a sentence supplies depth and distance information that, encoded as position representations, improves neural machine translation. The authors define absolute structural position as a word's tree depth measured from the main verb, and relative structural position through two rules for pairs on the same or different dependency edges. Across NIST Chinese-to-English and WMT14 English-to-German translation, the structural representations consistently beat both absolute and relative sequential position encodings, and linguistic probing shows the encoder retains more syntactic knowledge. If the claim is right, syntactic structure can be injected into Transformers through position encodings alone, without changing the attention mechanism.

What carries the argument

The machinery is two position formulas read off a dependency tree. Absolute structural position takes the dependency-path distance from a word to the main verb as its scalar position, $\mathrm{abs}_{\mathrm{stru}}(x_i)=\operatorname{distance}_{\mathrm{tree}}(x_i,\mathrm{origin})$. Relative structural position uses $\mathrm{abs}_{\mathrm{stru}}(x_i)-\mathrm{abs}_{\mathrm{stru}}(x_j)$ for words sharing a dependency edge, and for words on different edges uses $\operatorname{sign}(i-j)\cdot(\mathrm{abs}_{\mathrm{stru}}(x_i)+\mathrm{abs}_{\mathrm{stru}}(x_j))$, with clipping to limit the maximum relative distance. These scalars are passed through the same sinusoidal or learned embedding functions used for sequential positions, so the tree signal enters the attention computation without altering the self-attention mechanism.

What would settle it

Run the same Transformer-Big training setup with structural positions derived from a different dependency parser, or with the origin changed from the main verb to the tree root; if the reported BLEU gains over relative sequential position encoding disappear under either change, the improvement cannot be attributed to tree depth and distance as such.

Watch

Extended reading notes

Core claim

The central claim is that dependency-tree geometry is a useful position signal for self-attention, complementary to word order. Using tree depth as an absolute position and a hand-defined pairwise tree distance as a relative position, the authors report BLEU improvements over both the absolute and relative sequential position representations on NIST Chinese-to-English and WMT14 English-to-German translation. The gains hold on Transformer-Base and Transformer-Big, and the probing evaluation indicates the added signal is specifically syntactic.

Load-bearing premise

The load-bearing premise is that a dependency parse of the source sentence, combined with the two hand-written formulas for absolute depth and pairwise tree distance, is a faithful and useful enough encoding of structure that its errors do not cancel the gains.

Editorial extensions

If this is right

  • Adding absolute or relative structural position to a Transformer encoder without any sequential position raises NIST MT02 development BLEU from 28.33 to 35.43 or 34.23, showing the structural signal alone carries positional information.
  • On Transformer-Big, structural position encoding adds up to +0.93 BLEU on NIST Chinese-to-English and +0.61 BLEU on WMT14 English-to-German over the Transformer-Big baseline.
  • Structural position can be combined with either absolute or relative sequential encodings; combining it with relative sequential position gives the best results reported in the paper.
  • Probing tasks on the encoder outputs show the structural-position model scores higher on syntactic probes (65.87 vs 64.98) while keeping semantic probe performance, indicating the gain is tied to syntax rather than general representation quality.
  • The decoding-speed cost is marginal, so the augmentation is a practical change to the encoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors do not test is applying the same depth-and-distance encoding to decoder-side or cross-attention positions, where target-side parses could come from reference translations at training time or from a target-side parser at inference time.
  • Because the method relies only on depth and pairwise distance, it could be combined with unsupervised or automatically induced parse trees, potentially extending the benefit to languages without treebank-annotated parse data.
  • The two relative-position rules are one plausible encoding among many; comparing them with alternatives such as dependency-path length or lowest-common-ancestor distance would reveal whether the specific formula matters or any tree-geometric distance helps.
  • If structural position helps translation, it likely helps other tasks where explicit syntax matters, such as semantic role labeling or question answering, though the paper does not test those.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes augmenting Transformer self-attention with structural position representations derived from dependency trees. The absolute structural position is the tree depth of a word measured from the main verb, and the relative structural position is defined by two hierarchical rules (same-edge and cross-edge pairs). These structural encodings are integrated with the standard sequential absolute and relative position encodings on the encoder side. Experiments on NIST Chinese-to-English and WMT14 English-to-German report consistent BLEU gains over the sequential-only baselines, with bootstrap significance on many test sets, and probing experiments suggest that the encoder representations become more syntactically informative.

Significance. If the result holds, the paper makes a practical contribution: it shows that dependency-derived depth and distance signals can be injected into Transformer encoders through position representations, yielding consistent translation gains without changing the decoder. Strengths include evaluation on held-out test sets, bootstrap significance testing, comparison across two language pairs, and a probing analysis that corroborates the syntactic mechanism. The paper is not fully self-contained, however: the fusion function fabs is undefined, and the relative structural position formula in Eq. (6) does not compute the dependency path distance it claims. These issues leave the mechanism underlying the reported gains under-specified, though the central empirical claim remains plausible and the design can be repaired.

major comments (3)
  1. [Section 3.1, Eq. (6)] The relative structural position for words on different dependency edges is defined as fstru(i-j) * (absstru(xi) + absstru(xj)). This is not the dependency path distance described in Section 1 and Figure 1. In a rooted dependency tree, the path length between xi and xj is absstru(xi) + absstru(xj) - 2 * absstru(lca(xi, xj)); the lowest-common-ancestor term is missing. The encoding therefore overestimates cross-branch distances and mixes a sequential-order sign into a supposedly structural quantity. Since no experiment compares this proxy with the exact LCA-based tree distance or with another structural baseline, the reported BLEU improvements cannot be attributed specifically to structural distance. The authors should either correct Eq. (6) or explicitly justify the depth-sum/sign rule as a design choice and ablate it against the exact distance.
  2. [Section 3.2, Eq. (7) and footnote 1] The absolute-position fusion function fabs is left unspecified. Footnote 1 only says that fabs is nonlinear and that an alternative parameter-free element-wise addition gives a 0.28 BLEU improvement on the development set. Without a definition of fabs, the absolute structural variant is not reproducible, and the contribution of the nonlinear fusion to the results in Tables 1 and 2 cannot be assessed. The authors should specify fabs explicitly (for example, a concatenation followed by a linear layer and nonlinearity, or a gated sum) and state which variant was used for the reported results.
  3. [Section 5.2, Table 2] The central claim is that structural position representations consistently boost performance over both absolute and relative sequential position representations. The gains over the relative-sequential baseline are modest in places (e.g., +0.40 BLEU averaged over NIST sets and +0.29 BLEU on WMT14 En-De), and although bootstrap significance is reported for many individual test sets, some individual cells (for example, MT03 in the +Relative Sequential PE row) are not marked significant. The authors should report significance for the averaged gains and, ideally, provide multiple decoding runs or checkpoint averaging so that the reader can distinguish consistent improvements from noise.
minor comments (5)
  1. [Abstract and Section 5.1] The verb 'valid' is used where 'evaluate' or 'validate' is meant (e.g., 'we valid our proposed structural position encoding strategies'); please correct this throughout.
  2. [Section 5] The sentence 'obtain the structural structural absolute and relative position' contains a duplicated word; please fix the typo.
  3. [Section 3.1] The handling of BPE subwords and the end-of-sentence symbol is specified for absolute structural positions, but the corresponding handling for relative structural positions is not fully described. Please clarify how relative positions are assigned when one or both tokens are BPE subwords.
  4. [Table 1] The row encoding of the ablation is hard to read: the columns 'Sequential' and 'Structural' each contain 'Abs. Rel.' subcolumns, and the all-crossed first row is not immediately obvious. Please restructure the table so that each model configuration is unambiguous.
  5. [Section 3.1] The phrase 'set the the first larger integer than the max absolute structural position' is awkward and contains a typo ('the the'); please rephrase.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the structural encodings are hand-defined from an external parser and evaluated on held-out test sets.

full rationale

The paper's load-bearing chain is external and empirical, not circular. Dependency trees are produced by the Stanford parser (Klein and Manning, 2003), an external tool; absolute structural position is defined directly as tree depth to the main verb (Eq. 5); relative structural position is a stated hand-defined encoding (Eq. 6 and the two hierarchical rules); and the resulting representations are injected through the standard Shaw et al. (2018) relative-attention mechanism. The reported BLEU numbers are measured on held-out NIST and WMT14 test sets, so no fitted parameter is renamed as a prediction and no target quantity is defined in terms of the output. The clipping distance r=16 is taken from prior work, not tuned to these test sets. The skeptical concern that Eq. 6 encodes depth-sum rather than true LCA-based tree distance is a critique of whether the proposed proxy is faithful to the claimed linguistic mechanism; it is not a circularity, because the formula is presented as the authors' proposed encoding rather than derived from the experimental outcome. Similarly, the unspecified fusion function fabs in Eq. 7 and the reported 0.28 BLEU gain from element-wise addition are development-set observations and model-design details, not predictions forced by construction. Self-citations to the authors' other NMT works appear only as related work and do not carry the central argument, which rests on external parser input, external probing-task code, and independently evaluated translation quality. The derivation is therefore self-contained with respect to circularity, and any weaknesses are correctness or validity concerns rather than circular-reasoning defects.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the choice of dependency parsing, the hand-defined structural position formulas, and the heuristic handling of BPE and EOS tokens. The only explicit numeric free parameter is the clipping distance r=16. No new physical or conceptual entities are postulated.

free parameters (1)
  • Clipping distance r for relative structural positions = 16
    Set to 16 following Shaw et al. (2018), not tuned on the development set.
assumptions (6)
  • domain assumption Dependency tree depth and distance capture useful latent sentence structure.
    Invoked in Section 1 with reference to Hewitt and Manning (2019); if false, the structural positions carry no useful signal.
  • domain assumption The Stanford parser gives sufficiently accurate dependency trees for Chinese and English at training and inference time.
    Stated in Section 5; noisy parses would weaken the method.
  • domain assumption The main verb is a valid origin for absolute structural position.
    Defined in Section 3.1, Eq. 5; alternative roots could change all positions.
  • ad hoc to paper The relative structural position rules in Section 3.1 are a reasonable encoding of tree relationships.
    Rule 2 in particular, sign(i-j) times the sum of depths, is hand-defined and not derived or ablated.
  • ad hoc to paper BPE subwords share the original word's position and the EOS token gets max depth plus one.
    Stated in Section 3.1 as a heuristic; not validated separately.
  • standard math The Transformer baseline and its training recipe are correct as implemented in THUMT.
    Used in Section 5; the paper relies on existing implementations for the baseline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Attention with Structural Position Representations." pith.science (2026). https://pith.science/paper/5ANG4BBN

@misc{pith2026190900383,
  author       = {Pith},
  title        = {Pith review of: Self-Attention with Structural Position Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ANG4BBN}},
  note         = {Machine review of arXiv:1909.00383}
}
read the original abstract

Although self-attention networks (SANs) have advanced the state-of-the-art on various NLP tasks, one criticism of SANs is their ability of encoding positions of input words (Shaw et al., 2018). In this work, we propose to augment SANs with structural position representations to model the latent structure of the input sentence, which is complementary to the standard sequential positional representations. Specifically, we use dependency tree to represent the grammatical structure of a sentence, and propose two strategies to encode the positional relationships among words in the dependency tree. Experimental results on NIST Chinese-to-English and WMT14 English-to-German translation tasks show that the proposed approach consistently boosts performance over both the absolute and relative sequential position representations.

Figures

Figures reproduced from arXiv: 1909.00383 by the authors.

Figure 1
Figure 1. Illustration of (a) the standard sequential position encoding (Vaswani et al., 2017; Shaw et al., 2018), and (b) the proposed structural position encoding. The relative position in the example is for the word “talk”. it into queries Q ∈ R I×d , keys K ∈ R I×d , and values V ∈ R I×d : Q, K, V = XWQ, XWK, XWV ∈ R I×d , (1) where {WQ,WK,WV } ∈ R d×d are trainable parameters and d indicates the hidden size. The output s… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 24 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In the 7th Linguistic Annotation Workshop and Interoperability with Discourse

    work page 2013

  4. [4]

    Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. 2019. Attention augmented convolutional networks. arXiv

    2019

  5. [5]

    Thomas G Bever. 1970. The cognitive basis for linguistic structures. Cognition and the development of language, 279(362)

    1970

  6. [6]

    Noam Chomsky. 1965. Aspects of the Theory of Syntax, volume 11. MIT press

    1965

  7. [7]

    Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational linguistics, 29(4)

    work page 2003

  8. [8]

    Alexis Conneau, Germ \'a n Kruszewski, Guillaume Lample, Lo \"i c Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In ACL

    work page 2018

Show all 43 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL

    2019

  2. [10]

    Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. 2018. Exploiting deep representations for neural machine translation. In EMNLP

    2018

  3. [11]

    Ziyi Dou, Zhaopeng Tu, Xing Wang, Longyue Wang, Shuming Shi, and Tong Zhang. 2019. Dynamic layer aggregation for neural machine translation. In AAAI

    2019

  4. [12]

    Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. In NAACL

    2016

  5. [13]

    Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to parse and translate improves neural machine translation. In ACL

    2017

  6. [14]

    Jie Hao, Xing Wang, Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu. 2019 a . Multi-granularity self-attention for neural machine translation. In EMNLP

    2019

  7. [15]

    Jie Hao, Xing Wang, Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu. 2019 b . Towards better modeling hierarchical structure for self-attention with ordered neurons. In EMNLP

    2019

  8. [16]

    Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. 2019 c . Modeling recurrence for transformer. In NAACL

    2019

  9. [17]

    John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In NAACL

    2019

  10. [18]

    Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. 2017. Structured attention networks. In ICLR

    2017

  11. [19]

    Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In ACL

    2003

  12. [20]

    Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In EMNLP

    2004

  13. [21]

    Sandra K \"u bler, Ryan McDonald, and Joakim Nivre. 2009. Dependency parsing. Synthesis Lectures on Human Language Technologies, 1(1)

    2009

  14. [22]

    Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A Structured Self-attentive Sentence Embedding . In ICLR

    2017

  15. [23]

    Chunpeng Ma, Akihiro Tamura, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2019. Improving neural machine translation with neural syntactic distance. In NAACL)

    2019

  16. [24]

    Joakim Nivre. 2005. Dependency grammar and dependency parsing. MSI report, 5133(1959)

    1959

  17. [25]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL

    2002

  18. [26]

    a ckstr \

    Ankur Parikh, Oscar T \"a ckstr \"o m, Dipanjan Das, and Jakob Uszkoreit. 2016. A Decomposable Attention Model for Natural Language Inference . In EMNLP

    2016

  19. [27]

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations . In NAACL

    2018

  20. [28]

    Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In ICLR

    2019

  21. [29]

    Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019. Semantic neural machine translation using AMR . TACL

    2019

  22. [30]

    Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-Informed Self-Attention for Semantic Role Labeling . In EMNLP

    2018

  23. [31]

    Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In ACL

    2019

  24. [32]

    Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In ACL

    2015

  25. [33]

    Pasi Tapanainen and Timo Jarvinen. 1997. A non-projective dependency parser. In Fifth Conference on Applied Natural Language Processing

    1997

  26. [34]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need . In NIPS

    2017

  27. [35]

    Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, and Jingbo Zhu. 2018. Multi-layer representation fusion for neural machine translation. In COLING

    2018

  28. [36]

    Xing Wang, Zhengdong Lu, Zhaopeng Tu, Hang Li, Deyi Xiong, and Min Zhang. 2017. Neural machine translation advised by statistical machine translation. In AAAI

    2017

  29. [37]

    Xing Wang, Zhaopeng Tu, Longyue Wang, and Shuming Shi. 2019. Exploiting sentential context for neural machine translation. In ACL

    2019

  30. [38]

    Wei Wu, Houfeng Wang, Tianyu Liu, and Shuming Ma. 2018. Phrase-level self-attention networks for universal sentence encoding. In ACL

    2018

  31. [39]

    Wong, Lidia S

    Baosong Yang, Jian Li, Derek F. Wong, Lidia S. Chao, Xing Wang, and Zhaopeng Tu. 2019 a . Context-aware self-attention networks. In AAAI

    2019

  32. [40]

    Wong, Fandong Meng, Lidia S

    Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. 2018. Modeling localness for self-attention networks. In EMNLP

    2018

  33. [41]

    Wong, Lidia S

    Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019 b . Convolutional self-attention networks. In NAACL

    2019

  34. [42]

    Shan Yang, Heng Lu, Shiying Kang, Lei Xie, and Dong Yu. 2019 c . Enhancing hybrid self-attention structure with relative-position-aware bias for speech synthesis. In ICASSP

    2019

  35. [43]

    Jiacheng Zhang, Yanzhuo Ding, Shiqi Shen, Yong Cheng, Maosong Sun, Huanbo Luan, and Yang Liu. 2017. THUMT: An Open Source Toolkit for Neural Machine Translation . arXiv preprint arXiv:1706.06415

    2017 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.