Pith. sign in

REVIEW 4 major objections 5 minor 85 references

Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that syntax should enter a Transformer through its positional pathway, with the best injection site and coupling depending on the architecture, and that this improves syntactic generalization and downstream language…

desk verdict A careful empirical comparison of where syntax should enter transformer position pathways, but the core causal claim—that syntax, not the extra machinery around it, drives the gains—is not yet established. read the letter →

arxiv 2608.06111 v1 pith:UZPU2PG6 submitted 2026-08-06 cs.CL cs.AI

Haris Riaz , Hyungji Kim , Mihai Surdeanu This is my paper · ORCID
classification cs.CLcs.AI
keywords syntax-informedpositionalembeddingshexataggingdependencyparsingtransformerencodingrelativepositionrotarysyntacticgeneralizationGLUEbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SiPE injects a lightweight syntactic prior derived from dependency parses into the positional embedding pathway of Transformers. The paper claims that for autoregressive decoders with relative positional embeddings, the prior works best when multiplied into the relative-position term of the attention score, whereas for encoders it is best added directly to the input embedding. Pre-trained with SiPE, models improve on SyntaxGym by up to 10.3%, reduce perplexity by 9.0%, and raise GLUE scores by up to 8.2% over no-syntax baselines. Crucially, this is achieved by conditioning on a single parse at inference, rather than marginalizing over hundreds of parses, preserving most of the syntactic benefit at a fraction of the cost.

What carries the argument

The central object is the hexatag: a projective dependency parse is binarized into a binary head tree and linearized into two small tag vocabularies, two terminal tags and five non-terminal tags, assigned to each token. Two learned lookup tables, $E^T$ and $E^N$, map these tags to embedding vectors that are injected through the mask $m_p$; the key identity is Eq. 1, $m_p \cdot (E^T_{\tau_p} + E^N_{\nu_p})$, which supplies the syntactic prior at each word's first subword. For decoders, the load-bearing mechanism is the multiplicative position-pathway coupling, $\tilde{A}^{(\ell)}_{i,j,n} = AC^{(\ell)}_{i,j,n} + (1 + c^{(\ell)}_{i,j,n}) \cdot BD^{(\ell)}_{i,j,n}$, where $c$ is a tag-query alignment projected into head space; this gates the syntactic bias by the strength of the existing offset preference, so syntax sharpens an already-established positional signal rather than acting unconditionally.

What would settle it

Train the identical SiPE model with randomly shuffled or randomly sampled hexatags assigned to words, keeping all other training details the same, and compare SyntaxGym, perplexity, and GLUE against the real-tag SiPE model; if the random-tag model matches the real-tag model, the syntactic content of the tags is not the cause of the reported gains.

Watch

Extended reading notes

Core claim

SiPE establishes that syntactic structure can be delivered through the positional pathway instead of by constraining self-attention or complicating the architecture. For the relative-position decoder Transformer-XL, the strongest configuration multiplies a tag-query alignment into the existing position term, leaving the content term untouched; for encoders spanning absolute, relative, and rotary positional schemes, the simplest input-embedding addition is most reliable and composes with the native positional mechanism. The paper reports that these syntax-informed positional embeddings improve syntactic generalization on SyntaxGym and BLiMP while also reducing perplexity and improving GLUE, and that a single parse at inference time suffices to outperform parser-free syntax-injection methods and approach the syntactic gains of methods that marginalize over roughly 300 candidate parses.

Load-bearing premise

The measured gains are caused by the syntactic content of the hexatags rather than by the extra tag embeddings, the auxiliary tag-prediction loss, or the additional input conditioning, and the paper does not run a control with random or shuffled tags to show this.

Editorial extensions

If this is right

  • Syntactic supervision can be added to Transformers without modifying self-attention and without adding asymptotic inference cost beyond a single parser pass.
  • Architecture-specific placement matters: relative-position decoders should route syntax multiplicatively through the position term, while encoders should add it to the input embedding, and mixing injection sites can hurt.
  • Syntax-informed position embeddings improve not only syntactic benchmarks like SyntaxGym but also downstream GLUE tasks, including large-margin gains for a decoder-only model.
  • Injecting the syntactic prior from the earliest layer onward is important for decoders; skipping the first layer sharply reduces SyntaxGym accuracy.
  • Conditioning on a single parse instead of marginalizing over many parses moves the Pareto frontier between syntactic supervision and inference cost, making syntax injection practical for deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not run a control with random or shuffled hexatags, so its attribution of the gains to syntactic content rather than to the extra tag tables, tag-prediction loss, or input conditioning remains untested; a random-tag control would settle this.
  • Since richer dependency-relation labels did not help, the coarse directional tags may be supplying a useful inductive bias that is more about local attachment structure than about full relation semantics; testing tag vocabularies of intermediate granularity would probe this.
  • The decoder's weak rerouting of attention toward syntactically adjacent objects, despite large GLUE gains, suggests the syntactic prior may be absorbed into hidden representations rather than expressed in attention patterns; probing contextualized representations could localize where the benefit lives.
  • A direct extension would be to apply SiPE to a modern large decoder with rotary embeddings and test whether the multiplicative position-pathway result transfers, since the paper's decoder experiments use only Transformer-XL with relative positional embeddings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Syntax-informed Positional Embeddings (SiPE), which augment the absolute, relative, and rotary positional-encoding families with small learned embedding tables derived from hexatag dependency-parsing labels. For encoders (RoBERTa, DeBERTa-v3, ModernBERT) the prior is added to the input embedding, while for the Transformer-XL decoder the best configuration is a per-layer position-pathway projection that multiplicatively modulates the relative-position term of the attention score. The authors pretrain from scratch on a 50M-token WikiText slice and on BLLIP-LG, and report gains on BLiMP, SyntaxGym, BLLIP-LG perplexity, and GLUE, including the headline +10.3% SyntaxGym, -9.0% perplexity, and +8.2% GLUE macro-average improvements over a vanilla Transformer-XL.

Significance. If the results hold, the paper makes a practical contribution: a syntax-injection method that leaves self-attention and the rest of the architecture untouched, composes with all three dominant PE families, uses only a single parse at inference, and improves perplexity rather than degrading it. The strengths are real: same-architecture/same-data comparisons for the SiPE-versus-vanilla claims, three-seed standard deviations on GLUE, a systematic ablation over injection sites and coupling forms, a layerwise depth sweep, and unusually candid limitation statements. The central caveat is that the causal role of syntactic content has not been isolated from the added machinery, so the headline claims outrun the experimental design as currently presented.

major comments (4)
  1. [Section 3.1, Eqs. (1)-(2); Tables 2-6] The causal claim that the syntactic content of the hexatags drives the reported gains is not isolated. SiPE differs from the vanilla baseline in at least four ways simultaneously: the tag lookup tables E_T and E_N, the per-layer position-pathway projection W_E^(ℓ), the auxiliary tag-prediction loss, and, at evaluation time, conditioning on a parse. No control with randomly permuted or shuffled tags, nor a control with non-syntactic labels under identical capacity and auxiliary loss, is reported. Because the tag inventories are small (|T|=2, |N|=5), such controls are cheap and should be run; if a shuffled-tag model recovers most of the +10.3% SyntaxGym or +8.2% GLUE gains, the attribution to syntax would collapse.
  2. [Section 4.1 and Table 2] The headline 9.0% perplexity reduction compares p(x|T-hat(x)) for SiPE with the marginal p(x) for the vanilla Transformer-XL. The paper acknowledges the conditional protocol in Section 4.1, but the abstract and Table 2 present the two numbers as a direct improvement. A conditional distribution can lower perplexity simply because the conditioning parse carries information about upcoming tokens, even if the model's unconditional language modeling is no better. The same caveat affects SyntaxGym comparisons, since SiPE receives parse information for the test sentence while the baseline does not. Please report an estimate of the marginal p(x) for SiPE (e.g., by averaging over a small set of parses) or condition the vanilla baseline on an equally informative auxiliary signal, and state the comparison accordingly.
  3. [Section 6, Appendix A.2, Eq. (5)] The claim that the modification adds only O(1,000) parameters is inconsistent with the best PP-SiPE configuration, which uses a per-layer projection W_E^(ℓ) into the full model dimension (d_model x d_model per layer, Appendix A.2, Table 7). For a 16-layer Transformer-XL at d_model=768 this is roughly 9.4M parameters, several orders of magnitude larger than the stated O(1,000). This matters because the comparison to a vanilla model does not control for capacity. Please either correct the parameter count, use a capacity-matched baseline, or explicitly restrict the claim to the input-pathway variants.
  4. [Table 2 and Figure 1] The claim of a new Pareto frontier and of outperforming parser-free approaches (TreeReg, Tree-Planted Transformers) compares SiPE on Transformer-XL against baselines built on GPT-2 with different pretraining corpora and tokenizers. These are not same-architecture, same-data controls, so the relative deltas could reflect model family rather than the syntactic prior. Please add parser-free or syntax-free baselines on the same Transformer-XL backbone, or restrict the Pareto claim to a controlled comparison.
minor comments (5)
  1. [Section A.1] The sentence 'We adopt aassign hexatag to first-subword only strategy' contains a typo; it should read 'we adopt an assign-hexatag-to-first-subword-only strategy.'
  2. [Section 4.1] The text states that Transformer-XL is pretrained on BLLIP-LG following prior work and also that Transformer-XL is pretrained for 1M steps on the 50M-token WikiText slice; please state explicitly which pretraining setup underlies Table 2 and which underlies Table 6, since they appear to be different runs.
  3. [Table 5] The ModernBERT-large SiPE result (68.81 to 68.83 macro) shows no material improvement; given the general claim that gains extend to GLUE, this flat result deserves an explicit comment rather than being left in the table without discussion.
  4. [Section 6, Limitations] The phrase 'its unclear how the relative benefit ... will scale' should be 'it is unclear'.
  5. [Figure 1] The y-axis label 'SyntaxGym score' should state the unit ('%'), and the figure should indicate the backbone model for each method, since some points (TreeReg, TPT) use GPT-2 rather than Transformer-XL.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: SiPE's tag embeddings are learned and evaluated on held-out benchmarks, and no prediction reduces to a fitted parameter or to the paper's own definitions.

full rationale

SiPE's derivation chain is empirical rather than definitional: the tag embeddings in Eq. 1 are randomly initialized learned tables trained jointly with the LM objective, and the reported SyntaxGym, BLiMP, and GLUE numbers are measured on held-out benchmarks against baselines, not recovered from the training objective by construction. The multiplicative PP-SiPE coupling (Eqs. 5-6) is selected by comparing configurations on the same benchmarks, which is a model-selection risk, not a circular reduction. The perplexity comparison is conditional p(x|T-hat(x)) versus a marginal baseline, an apples-to-oranges protocol that the paper discloses in footnote 6, but this is a confound rather than a case where the prediction equals its input by definition. The only in-house component is the hexatagger, described as an implementation detail, and the hexatag formalism itself is cited to external prior work (Amini et al. 2023); no load-bearing self-citation, uniqueness theorem, or ansatz-smuggled-via-citation chain appears. The acknowledged limitations—parser error inheritance, KV-cache incompatibility, and small-scale English-only evaluation—concern generality and engineering, not circularity. I find no step in which a 'prediction' reduces to a fitted parameter or to the paper's own definitions.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the usefulness of hexatag syntax, on parser accuracy at inference, on the single-parse approximation, and on the representativeness of small-scale pretraining. No new physical or theoretical entities are introduced; tag embedding tables and projections are learned parameters that the method introduces.

free parameters (5)
  • Terminal tag embedding table E_T = learned during pretraining
    Maps the two terminal hexatag classes to vectors added to the input or attention score; central to the method.
  • Non-terminal tag embedding table E_N = learned during pretraining
    Maps the five non-terminal hexatag classes to vectors; central to the method.
  • Per-layer projection W_E^(l) for PP-SiPE = learned during pretraining
    Projects summed tag embeddings into head space for the tag-query alignment in Equations 5 and 12.
  • Per-layer phase-shift projection W_delta^(l) for RoPE = learned during pretraining
    Produces per-frequency rotary angle offsets in Equation 14.
  • Auxiliary tag-prediction heads = learned during pretraining
    Predict terminal and non-terminal tags at first-subword positions; part of the pretraining objective.
assumptions (5)
  • domain assumption Dependency parses (hexatags) provide a useful syntactic inductive bias for language modeling.
    This is the premise of the method; no random-tag control tests whether the tag content, rather than the extra signal, is what helps.
  • domain assumption Hexatagger errors are acceptable and the same parser can be used at inference.
    Stated in Limitations; the method inherits parser errors, and performance in low-parser-accuracy domains is untested.
  • domain assumption A single parse at inference approximates marginalization well enough to improve generalization.
    Central to the Pareto-frontier claim in Section 1 and Figure 1; not directly tested against marginalization in the same model.
  • ad hoc to paper Pretraining from scratch on a 50M-token WikiText slice is representative of real pretraining behavior.
    All models are trained under a small academic budget; scaling behavior is explicitly left open in Limitations.
  • domain assumption The auxiliary tag-prediction loss does not harm language modeling.
    Assumed by the joint objective; perplexity results suggest it does not, but no ablation removes only the auxiliary loss while keeping tags.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers." pith.science (2026). https://pith.science/paper/UZPU2PG6

@misc{pith2026260806111,
  author       = {Pith},
  title        = {Pith review of: Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZPU2PG6}},
  note         = {Machine review of arXiv:2608.06111}
}
abstract

Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to \textit{syntactic structure}. We introduce \textbf{S}yntax-\textbf{i}nformed \textbf{P}ositional \textbf{E}mbeddings (\textbf{SiPE}), which learns a lightweight syntactic prior from dependency parses during pretraining and injects it across all three dominant PE families (absolute, relative, rotary), for both encoders and decoders, leaving self-attention and the rest of the architecture untouched. We isolate \emph{where} and \emph{how} the prior should enter the model, and find it depends on the architecture: for autoregressive decoders that use relative PE, the prior is strongest when coupled multiplicatively with the relative-position term of the attention score, outperforming injection into the input embeddings, into self-attention, or into the positional and attention terms jointly---while for encoders it is best added directly to the input embeddings, composing with each encoder's native positional mechanism. We find that models pre-trained with SiPE improve on the SyntaxGym benchmark by up to $10.3\%$ while simultaneously reducing perplexity by $9.0\%$ over a base model with no syntactic supervision---a metric nearly every existing syntax-injection method instead degrades. Crucially, these gains extend beyond syntactic generalization: SiPE also improves real-world language understanding, raising scores on the GLUE benchmark by up to $8.2\%$ over a model trained without it. Unlike existing syntactic language models that marginalize over many parses at inference or discard syntax at runtime, SiPE conditions on a single parse, establishing a new Pareto frontier between syntactic supervision and inference cost.

Figures

Figures reproduced from arXiv: 2608.06111 by the authors.

Figure 1
Figure 1. Our Syntax-Informed Positional Embeddings (SiPE) move the Pareto frontier between syntactic super￾vision and inference cost. Each point plots a method’s Syn￾taxGym score ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Left: From bottom to top, the figure shows the correspondence between dependency arcs, binary head tree (BHT), and hexatags (left-shifted) for the sentence “she watches funny videos.” Right: Our prior-injection method for absolute positional embeddings (input-pathway injection). From bottom to top, each token (at its first subword; subsection A.1) selects one row from each of four lookup tables: the token embedding … view at source ↗
Figure 3
Figure 3. Layerwise SiPE injection sweep on Transformer￾XL. Injecting from layer k=1 (all layers) is best; skipping the first layer already drops SyntaxGym accuracy sharply, and later entry points degrade syntactic generalization further. under continued pre-training on BLLIP-LG af￾ter WikiText, rising to +1.87%, +3.06%, and +4.21% respectively, indicating the prior gener￾alizes beyond its original distribution rather than ov… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Verb→object attention on BLiMP (Warstadt et al., 2023) for RoBERTa-base vs. RoBERTa-base+SiPE (in￾put pathway injection), both pre-trained on the 50M-token WikiText slice (subsection 4.1). On the subset of Causative examples where SiPE assigns higher PLL to the grammat…
Figure 5
Figure 5. Figure 5: Verb→object attention on BLiMP for ModernBERT-base vs. ModernBERT-base+SiPE (input path￾way injection), both pre-trained on the 50M-token WikiText slice (subsection 4.1). On Causative examples where SiPE is correct and the base model is not, the SiPE model assigns high…
Figure 6
Figure 6. Figure 6: Verb→object attention on BLiMP for Transformer-XL vs. Transformer-XL+SiPE (positional pathway injection), both pre-trained on the 50M-token Wiki￾Text slice (subsection 4.1), averaged over all 16 layers. On Causative examples where SiPE assigns higher sentence log￾likel…
Figure 7
Figure 7. Figure 7: Average GLUE performance across prior-injection variants for RoBERTa-base, shown as the improvement [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: GLUE performance averaged across 3 random seeds [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Injection points for syntactic priors in a Transformer encoder. We study where and how to inject hexatag-derived priors (terminal ti , non-terminal ni , and optionally dependency-relation labels depreli ) into RoBERTa-base. Three architectural locations are considered:…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

85 extracted references · 31 canonical work pages

  1. [1]

    Afra Amini, Tianyu Liu, and Ryan Cotterell. 2023. https://arxiv.org/abs/2306.05477 Hexatagging: Projective dependency parsing as tagging . Preprint, arXiv:2306.05477

    work page Pith review arXiv 2023

  2. [2]

    Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics

    1998

  3. [4]

    Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. https://arxiv.org/abs/1901.02860 Transformer-xl: Attentive language models beyond a fixed-length context . Preprint, arXiv:1901.02860

    arXiv 2019

  4. [5]

    Marie-Catherine De Marneffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. Computational linguistics, 47(2):255--308

    2021

  5. [6]

    DeepSeek-AI. 2024. https://arxiv.org/abs/2412.19437 Deepseek-v3 technical report . Preprint, arXiv:2412.19437

    arXiv 2024

  6. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    arXiv 2019

  7. [8]

    Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others. 2024. https://api.semanticscholar.org/CorpusID:271571434 ...

    2024

  8. [9]

    Yinuo Guo, Zeqi Lin, Jian-Guang Lou, and Dongmei Zhang. 2020. Hierarchical poset decoding for compositional generalization in language. Advances in Neural Information Processing Systems, 33:6913--6924

    2020

Show all 85 references
  1. [10]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. https://arxiv.org/abs/2111.09543 Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing . Preprint, arXiv:2111.09543

    2023 arXiv

  2. [11]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://arxiv.org/abs/2006.03654 Deberta: Decoding-enhanced bert with disentangled attention . Preprint, arXiv:2006.03654

    2021 arXiv

  3. [12]

    Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger P. Levy. 2020. https://arxiv.org/abs/2005.03692 A systematic assessment of syntactic generalization in neural language models . Preprint, arXiv:2005.03692

    2020 arXiv

  4. [13]

    Hu, Jackson Petty, Chuan Shi, William Merrill, and Tal Linzen

    Michael Y. Hu, Jackson Petty, Chuan Shi, William Merrill, and Tal Linzen. 2025. https://arxiv.org/abs/2502.19249 Between circuits and chomsky: Pre-pretraining on formal languages imparts linguistic biases . Preprint, arXiv:2502.19249

    2025 arXiv

  5. [14]

    Gemma Team Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram'e, Morgane Rivi \`e re, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casb...

    2025 arXiv

  6. [15]

    Paul R Kingsbury and Martha Palmer. 2002. From treebank to propbank. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 1989--1993

    2002

  7. [16]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  8. [18]

    Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. 2023. https://arxiv.org/abs/2310.19089 Pushdown layers: Encoding recursive structure in transformer language models . Preprint, arXiv:2310.19089

    2023 arXiv

  9. [19]

    Manning, and Shikhar Murty

    Ananjan Nandi, Christopher D. Manning, and Shikhar Murty. 2025. https://arxiv.org/abs/2411.18885 Sneaking syntax into transformer language models with tree regularization . Preprint, arXiv:2411.18885

    2025 arXiv

  10. [20]

    Yutaro Omote, Akihiro Tamura, and Takashi Ninomiya. 2019. https://doi.org/10.26615/978-954-452-056-4_099 Dependency-based relative positional encoding for transformer nmt . In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP ...

    2019 doi

  11. [22]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  12. [25]

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. https://arxiv.org/abs/1803.02155 Self-attention with relative position representations . Preprint, arXiv:1803.02155

    2018 arXiv

  13. [27]

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. https://arxiv.org/abs/2104.09864 Roformer: Enhanced transformer with rotary position embedding . Preprint, arXiv:2104.09864

    2023 arXiv

  14. [29]

    Thinking Machines Lab . 2026. Inkling: Our open-weights model. https://thinkingmachines.ai/news/introducing-inkling/. Blog post; model weights at https://huggingface.co

    2026

  15. [30]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  16. [31]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. https://arxiv.org/abs/1804.07461 Glue: A multi-task benchmark and analysis platform for natural language understanding . Preprint, arXiv:1804.07461

    2019 arXiv

  17. [32]

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. https://arxiv.org/abs/2412.13663 Smarter, better, f...

    2024 arXiv

  18. [33]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2023. https://arxiv.org/abs/1912.00582 Blimp: The benchmark of linguistic minimal pairs for english . Preprint, arXiv:1912.00582

    2023 arXiv

  19. [34]

    Yikuan Xie, Wenyong Wang, Mingqian Du, and Qing He. 2021. https://aclanthology.org/2021.ranlp-1.172/ Transformer with syntactic position encoding for machine translation . In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2...

    2021

  20. [38]

    Yida Zhao, Chao Lou, and Kewei Tu. 2024. https://arxiv.org/abs/2407.17406 Dependency transformer grammars: Integrating dependency structures into transformer language models . Preprint, arXiv:2407.17406

    2024 arXiv

  21. [39]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  22. [40]

    Publications Manual , year = "1983", publisher =

    1983

  23. [41]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    1981

  24. [42]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  25. [43]

    Dan Gusfield , title =. 1997

    1997

  26. [44]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  27. [45]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  28. [46]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  29. [47]

    2019 , eprint=

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , author=. 2019 , eprint=

    2019

  30. [48]

    2023 , eprint=

    Hexatagging: Projective Dependency Parsing as Tagging , author=. 2023 , eprint=

    2023

  31. [49]

    2025 , eprint=

    Sneaking Syntax into Transformer Language Models with Tree Regularization , author=. 2025 , eprint=

    2025

  32. [50]

    2025 , eprint=

    Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases , author=. 2025 , eprint=

    2025

  33. [51]

    2024 , eprint=

    Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models , author=. 2024 , eprint=

    2024

  34. [52]

    Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale , volume=

    Sartran, Laurent and Barrett, Samuel and Kuncoro, Adhiguna and Stanojević, Miloš and Blunsom, Phil and Dyer, Chris , year=. Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale , volume=. doi:10.1162/tacl_a_00526 , journal=

    doi

  35. [53]

    2018 , eprint=

    Self-Attention with Relative Position Representations , author=. 2018 , eprint=

    2018

  36. [54]

    Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019) , pages =

    Yutaro Omote and Akihiro Tamura and Takashi Ninomiya , title =. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019) , pages =. 2019 , doi =

    2019

  37. [55]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    Hao Fei and Yafeng Ren and Donghong Ji , title =. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =. 2020 , doi =

    2020

  38. [56]

    Dependency Parsing

    Kübler, Sandra and McDonald, Ryan and Nivre, Joakim. Dependency Parsing. 2009. doi:10.2200/S00169ED1V01Y200903HLT002

    2009 doi

  39. [57]

    2019 , eprint=

    RoBERTa: A Robustly Optimized BERT Pretraining Approach , author=. 2019 , eprint=

    2019

  40. [58]

    2021 , eprint=

    DeBERTa: Decoding-enhanced BERT with Disentangled Attention , author=. 2021 , eprint=

    2021

  41. [59]

    2023 , eprint=

    DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing , author=. 2023 , eprint=

    2023

  42. [60]

    Structural Guidance for Transformer Language Models

    Qian, Peng and Naseem, Tahira and Levy, Roger and Fernandez Astudillo, Ram \'o n. Structural Guidance for Transformer Language Models. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural...

    2021 doi

  43. [61]

    Composition, Attention, or Both? , url=

    Yoshida, Ryo and Oseki, Yohei , year=. Composition, Attention, or Both? , url=. doi:10.18653/v1/2022.findings-emnlp.428 , booktitle=

    2022 doi

  44. [62]

    2023 , eprint=

    Pushdown Layers: Encoding Recursive Structure in Transformer Language Models , author=. 2023 , eprint=

    2023

  45. [63]

    Language , pages=

    Structure dependence in grammar formation , author=. Language , pages=. 1987 , publisher=

    1987

  46. [64]

    Topics in cognitive science , volume=

    A model of language processing as hierarchic sequential prediction , author=. Topics in cognitive science , volume=. 2013 , publisher=

    2013

  47. [65]

    Proceedings of the 56th annual meeting of the association for computational linguistics (Volume 1: Long Papers) , pages=

    Finding syntax in human encephalography with beam search , author=. Proceedings of the 56th annual meeting of the association for computational linguistics (Volume 1: Long Papers) , pages=

  48. [66]

    Advances in Neural Information Processing Systems , volume=

    Hierarchical poset decoding for compositional generalization in language , author=. Advances in Neural Information Processing Systems , volume=

  49. [67]

    2023 , eprint=

    RoFormer: Enhanced Transformer with Rotary Position Embedding , author=. 2023 , eprint=

    2023

  50. [68]

    2024 , eprint=

    Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference , author=. 2024 , eprint=

    2024

  51. [69]

    2019 , eprint=

    GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , author=. 2019 , eprint=

    2019

  52. [70]

    2023 , eprint=

    BLiMP: The Benchmark of Linguistic Minimal Pairs for English , author=. 2023 , eprint=

    2023

  53. [71]

    and Kirchhoff, Katrin , year=

    Salazar, Julian and Liang, Davis and Nguyen, Toan Q. and Kirchhoff, Katrin , year=. Masked Language Model Scoring , url=. doi:10.18653/v1/2020.acl-main.240 , booktitle=

    2020 doi

  54. [72]

    2023 , eprint=

    A Better Way to Do Masked Language Model Scoring , author=. 2023 , eprint=

    2023

  55. [73]

    arXiv preprint arXiv:1609.07843 , year=

    Pointer Sentinel Mixture Models , author=. arXiv preprint arXiv:1609.07843 , year=

    arXiv

  56. [74]

    2019 , eprint=

    Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context , author=. 2019 , eprint=

    2019

  57. [75]

    Transformer with Syntactic Position Encoding for Machine Translation

    Xie, Yikuan and Wang, Wenyong and Du, Mingqian and He, Qing. Transformer with Syntactic Position Encoding for Machine Translation. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021). 2021

    2021

  58. [76]

    doi:10.35111/fwew-da58 , note =

    Charniak, Eugene and Blaheta, Don and Ge, Niyu and Hall, Keith and Hale, John and Johnson, Mark , year =. doi:10.35111/fwew-da58 , note =

    doi

  59. [77]

    2020 , eprint=

    A Systematic Assessment of Syntactic Generalization in Neural Language Models , author=. 2020 , eprint=

    2020

  60. [78]

    2023 , eprint=

    On the Challenges of Fully Incremental Neural Dependency Parsing , author=. 2023 , eprint=

    2023

  61. [79]

    An Efficient Algorithm for Projective Dependency Parsing

    Nivre, Joakim. An Efficient Algorithm for Projective Dependency Parsing. Proceedings of the Eighth International Conference on Parsing Technologies. 2003

    2003

  62. [80]

    ArXiv , year=

    Arc-swift: A Novel Transition System for Dependency Parsing , author=. ArXiv , year=

  63. [81]

    Computational linguistics , volume=

    Universal dependencies , author=. Computational linguistics , volume=

  64. [82]

    COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics , year=

    The berkeley framenet project , author=. COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics , year=

    1998

  65. [83]

    Proceedings of the Language Resources and Evaluation Conference (LREC) , pages=

    From TreeBank to PropBank , author=. Proceedings of the Language Resources and Evaluation Conference (LREC) , pages=

  66. [84]

    Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision , url=

    Yoshida, Ryo and Someya, Taiga and Oseki, Yohei , year=. Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision , url=. doi:10.18653/v1/2024.findings-acl.303 , booktitle=

    2024 doi

  67. [85]

    S entence P iece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

    Kudo, Taku and Richardson, John. S entence P iece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2018. doi:10.18653/...

    2018 doi

  68. [86]

    Effective Inference for Generative Neural Parsing

    Stern, Mitchell and Fried, Daniel and Klein, Dan. Effective Inference for Generative Neural Parsing. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. doi:10.18653/v1/D17-1178

    2017 doi

  69. [87]

    Variable beam search for generative neural parsing and its relevance for the analysis of neuro-imaging signal

    Crabb \'e , Benoit and Fabre, Murielle and Pallier, Christophe. Variable beam search for generative neural parsing and its relevance for the analysis of neuro-imaging signal. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Int...

    2019 doi

  70. [88]

    2024 , url=

    The Llama 3 Herd of Models , author=. 2024 , url=

    2024

  71. [89]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 Technical Report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  72. [90]

    2024 , eprint=

    DeepSeek-V3 Technical Report , author=. 2024 , eprint=

    2024

  73. [91]

    ArXiv , year=

    Gemma 3 Technical Report , author=. ArXiv , year=

  74. [92]

    Language Models are Unsupervised Multitask Learners , author=

  75. [93]

    BERT Rediscovers the Classical NLP Pipeline

    Tenney, Ian and Das, Dipanjan and Pavlick, Ellie. BERT Rediscovers the Classical NLP Pipeline. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019. doi:10.18653/v1/P19-1356

    2019 doi

  76. [94]

    Linguistically-Informed Self-Attention for Semantic Role Labeling

    Strubell, Emma and Verga, Patrick and Andor, Daniel and Weiss, David and McCallum, Andrew. Linguistically-Informed Self-Attention for Semantic Role Labeling. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018. doi:10.18653/v1/D18-1548

    2018 doi

  77. [95]

    2026 , howpublished =

    Inkling: Our Open-Weights Model , author =. 2026 , howpublished =

    2026

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.