Pith. sign in

REVIEW 5 major objections 7 minor 36 references

A Systematic Study of Compositional Syntactic Transformer Language Models

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A syntactic Transformer language model's behavior is set by four binary design toggles, and the best combination—binary trees, bottom-up linearization, an external composition function, and no sub-constituent mask—delivers strong…

desk verdict Useful systematic comparison of compositional SLMs, but the downstream claims over GPT2-token are confounded by decoding. read the letter →

arxiv 2506.22978 v1 pith:IRLTX6T6 submitted 2025-06-28 cs.CL cs.AI

Yida Zhao , Hao Xve , Xiang Hu , Kewei Tu This is my paper · ORCID
classification cs.CLcs.AI
keywords syntacticlanguagemodelscompositionalSLMsconstituencyparsetreesexplicitcompositionsub-constituentmaskinggeneralizationword-synchronousbeamsearchdesignchoices
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks which design choices matter when a Transformer language model is given explicit constituency syntax, and it answers by building one unified space of compositional syntactic language models. Every model in the space jointly predicts a sentence and a linearized parse tree, and explicitly composes sub-constituent representations bottom-up; the choices are binary vs non-binary trees, top-down vs bottom-up linearization, internal vs external composition, and masking vs not masking composed sub-constituents. Across all sixteen variants, the paper finds that explicit syntax and composition do not help raw language-model perplexity (the reported numbers are upper bounds, so this is not conclusive), but the best variants improve syntactic generalization, summarization, and dialogue over a plain Transformer. The practical message is a set of design recommendations: avoid sub-constituent masking in most settings, and if speed matters use binary trees with an external composition function. A sympathetic reader would take the paper's core claim to be that these four toggles, not just the presence of syntax, determine whether compositional SLMs pay off within the tested scope of unlabeled constituency trees and a single Transformer backbone.

What carries the argument

The load-bearing object is the action sequence that linearizes a sentence together with its constituency tree, defining an autoregressive joint distribution over sentences and trees. Around that sequence the paper organizes every design choice: tree binarization (non-binary vs left-binarized), linearization order (top-down vs bottom-up), the composition function that folds sub-constituent representations into a single constituent representation (an extra Transformer module for external composition, or a masked attention step inside the main Transformer for internal composition), and whether composed sub-constituents stay visible to later attention (no-mask vs mask). The framework turns those four binary choices into sixteen named model instances, and the empirical comparison across those instances is what carries the recommendations.

What would settle it

Train the same model variants on gold human-annotated constituency trees from the same corpus, or on silver trees produced by two or three different parsers, and rerun the syntactic-generalization suites; if the ranking of the sixteen variants changes or the best variants no longer beat a plain Transformer, the paper's conclusions depend on the parser's tree distribution rather than on the design choices themselves.

Watch

Extended reading notes

Core claim

The central claim is that a compositional syntactic language model can be decomposed into four binary design decisions, and that the right combination of those decisions—not syntax alone—determines whether the model helps. The paper proposes a framework whose sixteen instances include three prior models as special cases and thirteen new variants, then evaluates all of them on language modeling, the syntactic-generalization test suites, summarization, dialogue, and inference time. The empirical pattern is consistent: no compositional SLM beats a plain Transformer on document-level perplexity (though the perplexity values are upper bounds, so the comparison is not decisive), while the best configurations substantially improve syntactic generalization and can improve downstream generation tasks. Explicit composition is shown to be helpful for syntactic generalization relative to a syntax-only SLM without composition, but only when the composition function can handle the tree shape: variants that compose non-binary trees with a small external function fail, whereas external functions on binary trees occupy the top scores. From these results the paper recommends leaving composed sub-constituents unmasked unless syntactic generalization is the sole goal, and choosing binary trees plus an external composition function when inference efficiency is the priority.

Load-bearing premise

The load-bearing premise is that the silver parse trees from one off-the-shelf CRF parser are trustworthy enough to serve as ground truth for both training and evaluation; if those trees carry parser-specific biases, the syntactic-generalization gains attributed to architectural choices could instead reflect the parser's distribution.

Editorial extensions

If this is right

  • Sub-constituent masking should be avoided except when syntactic generalization is the only objective, because unmasked variants are consistently better at language modeling and downstream generation and are faster to run.
  • If inference speed is the priority, configure the model with binary parse trees and an external composition function: this combination is several times faster than internal composition while keeping competitive scores.
  • Binary trees should be paired with bottom-up linearization, and non-binary trees with top-down linearization, for both performance and efficiency.
  • Modeling non-binary trees with an external composition function is a configuration to avoid, since the small external module fails to compose variable numbers of sub-constituents.
  • Explicit composition is not necessary for good language-model perplexity, but it is valuable for syntactic generalization; a syntax-only model without composition sits between a plain Transformer and the best compositional variants on that task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's rankings are trained and evaluated on silver parse trees from a single CRF parser; a natural extension is to ask whether the same variant rankings survive with gold human-annotated trees or trees from several parsers, since the parser's bias could favour certain linearizations or composition functions.
  • The failure of external composition on non-binary trees is attributed by the authors to the small composition module and the lack of a direct training signal for composition; this suggests a testable repair—larger external modules or auxiliary composition losses might close the gap without losing the efficiency advantage.
  • The four-binary-choice scheme defines a design space that generalizes beyond constituency trees; the same experimental template could rank design decisions for dependency-based syntactic language models, labeled trees, or other structured inductive biases.
  • A concrete next experiment suggested by the paper's own efficiency data: measure whether the speed advantage of external composition persists when FlashAttention-style kernels are adapted to its causal mask, since the paper notes most SLM attention patterns block such optimizations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes a unified framework for compositional syntactic Transformer language models, parameterized by four binary design choices: binarized vs non-binary trees (Bi/Nb), top-down vs bottom-up linearization (Dn/Up), internal vs external composition functions (In/Ex), and sub-constituent masking vs no masking (M/Nm). Sixteen variants are instantiated and evaluated on document-level language modeling (BLLIP-LG), syntactic generalization (SG), Xsum summarization, DailyDialog dialogue, and inference efficiency. The main empirical conclusions are that no-mask variants generally do better on downstream tasks and efficiency, binary trees with an external composition function are an efficient and competitive configuration, and top-performing variants improve over a token-level GPT-2 baseline on some tasks. The paper explicitly discloses that SLM perplexities are upper bounds and releases code.

Significance. If the findings are robust, the paper provides a useful mapping of design choices for compositional SLMs and a reusable codebase, unifying existing models such as Transformer Grammars, Composition Attention Grammars, and GPST under one framework. Strengths include the clean taxonomy, the transparent reporting of perplexity as upper bounds, the inclusion of inference-time and forward-call measurements, and the public release of code. The main limitations are the unmatched decoding procedures in the baseline comparisons, the reliance on a single training run for most configurations, and the use of one off-the-shelf CRF parser for both training trees and evaluation trees, which collectively weaken the headline claims about downstream gains and syntactic generalization.

major comments (5)
  1. [Section 3.3.1 / Appendix H] The comparison between SLMs and GPT2-token in Table 2 is confounded by decoding. As stated in Section 3.3.1 and Appendix H, SLMs are evaluated with word-synchronous beam search while GPT2-token is decoded with top-k random sampling (k=2). Beam search is generally much stronger than sampling for ROUGE-style metrics, so the reported gains (e.g., Xsum R-A 18.82 to 20.33; DailyDialog VG 10.38 to 10.59) cannot be attributed to syntactic composition over a Transformer baseline. The discussion in Section 3.3.1 lists two explanations for the SLM gains but omits this search advantage. The within-SLM variant comparisons are unaffected, but the abstract's and conclusion's claims that compositional SLMs improve summarization and dialogue performance over the Transformer LM baseline need to be supported by matched decoding (e.g., beam search for GPT2-token, or sampling for SLMs as an upper/lower bound), or rephrased as conditional on decoding method.
  2. [Appendix B / Tables 1-3] Except for Bi-Up-Ex-Nm (Appendix B), every configuration in Tables 1-3 is a single training run. Differences that drive the recommendations are often small: SG scores of 80.1 vs 80.2 or 80.9 in Table 1, Xsum R-A 20.33 vs 20.29 in Table 2, and perplexity gaps around 1-2 points. With no variance estimates or significance tests for the remaining variants, conclusions such as 'Bi-#-#-M always performs better than Bi-#-#-Nm' (Section 3.2) and the design recommendations in Section 3.5 rest on effect sizes that could be within training noise. Reporting repeated runs for the main contrasts (mask vs no-mask, internal vs external) or providing confidence intervals would be necessary to support the comparative claims.
  3. [Section 3, Dataset / Section 3.2] The use of one off-the-shelf CRF parser for both training trees and evaluation trees is load-bearing for the syntactic generalization claim. The paper states that 'All the silver trees parsed or sampled in the rest of the experiments are also produced with the same parser,' which appears to include the SG stimuli. If the SG test items are parsed with the same parser that generated the training trees, a model can score high on SG by learning parser-specific bracketing conventions rather than human syntactic generalizations. This weakens the comparison against GPT2-token on SG in Table 1. The authors should report results with gold trees or with a second parser on at least a subset of the SG suites, or otherwise account for the influence of parser identity.
  4. [Section 3.3.1] Only eight of the sixteen compositional SLMs are evaluated on Xsum and DailyDialog; the other eight are excluded after observing their language modeling and SG results (four Nb-Ex variants and four Bi-Dn variants). This selection is post hoc and can make the downstream results appear stronger than a full comparison. In addition, the abstract's claim of a 'comprehensive empirical evaluation of all the variants' across summarization and dialogue is not exact. The paper should either include all variants in the downstream tables or explicitly state that the downstream conclusions are conditional on the pre-selection based on the earlier tasks.
  5. [Section 3.5, Recommendation (i)] Recommendation (i) is not consistent with the SG results in Table 1. The recommendation says sub-constituent masking has only a 'potentially small disadvantage' in syntax-focused tasks, but for all four binary configurations masking improves SG: Bi-Up-Ex goes from 80.1 to 82.4, Bi-Up-In from 77.5 to 79.7, Bi-Dn-Ex from 80.2 to 80.9, and Bi-Dn-In from 79.4 to 80.9. Section 3.2 itself states that 'Bi-#-#-M always performs better than Bi-#-#-Nm.' The recommendation should acknowledge the consistent SG advantage of masking for binary trees and restrict the 'avoid masking' advice to non-binary configurations or to downstream tasks and efficiency.
minor comments (7)
  1. [Section 3.4] 'Casual mask' should be 'causal mask' in the sentence about scaled_dot_product_attention.
  2. [Section 2.3] In the description of the external composition function, the symbol 'αk' should be 'ak' for consistency with the rest of the paper.
  3. [Appendix C] The appendix title 'Hyperparameters Selection' should be 'Hyperparameter Selection'.
  4. [Table 4] The column labeled 'R-A VG (std)' is ambiguous; it contains a single value (20.33 (0.06)) and the reader has to infer that it refers to Xsum R-A. Please label the columns explicitly.
  5. [Appendix E] 'Six circuits' should be 'six phenomena' or 'six test suites' to match the terminology used in Section 3.2.
  6. [Section 3.3.1] The phrase 'apply the word-synchronous beam search to top-k random sampling' conflates two distinct inference algorithms. Please state the exact procedure (e.g., whether the SLMs use a beam of size 2 with k=2 or something else) so the decoding mismatch is explicit.
  7. [Limitations] The Limitations section discusses the perplexity upper bound and FlashAttention but does not mention the decoding mismatch with GPT2-token or the parser-specific silver-tree concern; adding these would make the limitations more complete.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper is an empirical comparison, and its central recommendations are not built into the definitions or fit parameters.

full rationale

This paper proposes a framework and compares instantiations empirically; it does not claim a mathematical derivation from first principles. The central quantities—perplexity, syntactic generalization scores, ROUGE, and dialogue metrics—are measured on held-out data and baselines, not constructed from fitted parameters. The only notable self-citations are (i) adopting the external composition function from the authors' own GPST work (Hu et al., 2024) and (ii) using the SG benchmark and BLLIP-LG splits from Hu et al. (2020), but in both cases the models are trained and evaluated independently and the cited works provide baselines or implementations, not the paper's conclusions. The paper's stated recommendations, such as preferring external composition for efficiency or avoiding sub-constituent masking, follow from the reported comparisons rather than from the definitions. The reliance on silver trees from a single CRF parser is a data-quality and generalization concern, not a circularity: the parser's output is external input, and the paper does not define success in terms of that parser's own predictions. Likewise, the differing decoding procedures for GPT2-token versus SLMs in downstream tasks is a potential confound but not a case where the result is equivalent to its inputs by construction. Overall, the derivation chain is self-contained enough that no specific circular reduction can be exhibited.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claims rest on standard empirical assumptions about parser quality, approximate inference, and evaluation validity, plus one tuned hyperparameter (pc). No new entities are introduced.

free parameters (1)
  • pc (maximum number of consecutive opening-nonterminal actions) = 3 for SG, 5 for summarization
    Tuned on 10 short sentences from the BLLIP-LG training set and then used for evaluation, which is a fitted hyperparameter rather than a fixed rule. Location: Appendix C.
assumptions (3)
  • domain assumption The CRF parser (Zhang et al., 2020) produces adequate silver trees for training and evaluation.
    All models are trained and evaluated on these trees; if they are systematically biased, the SG results would reflect parser bias rather than architectural differences. Location: Section 3, Dataset and Preprocessing.
  • domain assumption The approximate marginalization over 300 sampled trees yields a reliable lower bound for sentence probability.
    The paper acknowledges this gives an upper bound of perplexity, but assumes the bound is tight enough for fair model comparison. Location: Section 3.1.
  • domain assumption The SG test suites (Hu et al., 2020) validly measure syntactic generalization.
    These suites are used as the primary metric for syntactic ability; if they do not measure what is claimed, the main positive result is undermined. Location: Section 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Systematic Study of Compositional Syntactic Transformer Language Models." pith.science (2026). https://pith.science/paper/IRLTX6T6

@misc{pith2026250622978,
  author       = {Pith},
  title        = {Pith review of: A Systematic Study of Compositional Syntactic Transformer Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRLTX6T6}},
  note         = {Machine review of arXiv:2506.22978}
}
read the original abstract

Syntactic language models (SLMs) enhance Transformers by incorporating syntactic biases through the modeling of linearized syntactic parse trees alongside surface sentences. This paper focuses on compositional SLMs that are based on constituency parse trees and contain explicit bottom-up composition of constituent representations. We identify key aspects of design choices in existing compositional SLMs and propose a unified framework encompassing both existing models and novel variants. We conduct a comprehensive empirical evaluation of all the variants in our framework across language modeling, syntactic generalization, summarization, dialogue, and inference efficiency. Based on the experimental results, we make multiple recommendations on the design of compositional SLMs. Our code is released at https://github.com/zhaoyd1/compositional_SLMs.

Figures

Figures reproduced from arXiv: 2506.22978 by the authors.

Figure 1
Figure 1. An example sentence, its binary and non [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The Transformer-based external composition [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples of different mask patterns combined with different composition functions. We use [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: SG scores on each syntactic phenomenon. Model Time_Per_Token (ms) GPT2-token 5.6 GPT2-tree 50.9 Bi-Up-Ex-Nm 26.1 Bi-Up-Ex-M 27.8 Bi-Up-In-Nm 57.1 Bi-Up-In-M 64.6 Nb-Up-In-Nm 65.5 Nb-Up-In-M 71.1 Nb-Dn-In-Nm 70.2 Nb-Dn-In-M 84.1 [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 13 canonical work pages

  1. [1]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  2. [2]

    Eugene Charniak, Don Blaheta, Niyu Ge, Keith Hall, John Hale, and Mark Johnson. 2000. Bllip 1987-89 wsj corpus release 1. Linguistic Data Consortium, 36

    work page 2000

  3. [3]

    Ciprian Chelba. 1997. https://doi.org/10.3115/976909.979681 A structured language model . In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the E uropean Chapter of the Association for Computational Linguistics , pages 498--500, Madrid, Spain. Association for Computational Linguistics

    work page arXiv 1997

  4. [4]

    Do Kook Choe and Eugene Charniak. 2016. https://doi.org/10.18653/v1/D16-1257 Parsing as language modeling . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2331--2336, Austin, Texas. Association for Computational Linguistics

    doi:10.18653/v1/d16-1257 2016

  5. [5]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R'e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R'e. 2022. https://api.semanticscholar.org/CorpusID:249151871 Flashattention: Fast and memory-efficient exact attention with io-awareness . ArXiv, abs/2205.14135

    arXiv 2022

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  7. [7]

    Brian DuSell and David Chiang. 2021. Learning hierarchical structures with differentiable nondeterministic stacks. arXiv preprint arXiv:2109.01982

    work page Pith review arXiv 2021

  8. [8]

    Brian DuSell and David Chiang. 2023. Stack attention: Improving the ability of transformers to model hierarchical patterns. arXiv preprint arXiv:2310.01749

    work page Pith review arXiv 2023

Show all 36 references
  1. [9]

    Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. https://doi.org/10.18653/v1/N16-1024 Recurrent neural network grammars . In Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Lang...

    2016 doi

  2. [10]

    Martin BH Everaert, Marinus AC Huybregts, Noam Chomsky, Robert C Berwick, and Johan J Bolhuis. 2015. Structures, not strings: Linguistics as part of the cognitive sciences. Trends in cognitive sciences, 19(12):729--743

    2015

  3. [11]

    James Henderson. 2004. https://doi.org/10.3115/1218955.1218968 Discriminative training of a neural network statistical parser . In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics ( ACL -04) , pages 95--102, Barcelona, Spain

    2004

  4. [12]

    Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. https://doi.org/10.18653/v1/2020.acl-main.158 A systematic assessment of syntactic generalization in neural language models . In Proceedings of the 58th Annual Meeting of the Association for Computationa...

    2020 doi

  5. [13]

    Xiang Hu, Pengyu Ji, Qingyang Zhu, Wei Wu, and Kewei Tu. 2024. https://doi.org/10.18653/v1/2024.acl-long.145 Generative pretrained structured transformers: Unsupervised syntactic language models at scale . In Proceedings of the 62nd Annual Meeting of the Association for Comput...

    2024 doi

  6. [14]

    Armand Joulin and Tomas Mikolov. 2015. Inferring algorithmic patterns with stack-augmented recurrent nets. Advances in neural information processing systems, 28

    2015

  7. [15]

    Yoon Kim, Carl Denton, Luong Hoang, and Alexander M. Rush. 2017. https://openreview.net/forum?id=HkE0Nvqlg Structured attention networks . In International Conference on Learning Representations

    2017

  8. [16]

    Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and G \'a bor Melis. 2019. https://doi.org/10.18653/v1/N19-1114 Unsupervised recurrent neural network grammars . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computat...

    2019 doi

  9. [17]

    Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. https://aclanthology.org/I17-1099/ D aily D ialog: A manually labelled multi-turn dialogue dataset . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume ...

    2017

  10. [18]

    Chin-Yew Lin and Eduard Hovy. 2003. https://aclanthology.org/N03-1020/ Automatic evaluation of summaries using n-gram co-occurrence statistics . In Proceedings of the 2003 Human Language Technology Conference of the North A merican Chapter of the Association for Computational ...

    2003

  11. [19]

    Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher Manning. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.195 Pushdown layers: Encoding recursive structure in transformer language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural L...

    2023 doi

  12. [20]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. https://doi.org/10.18653/v1/D18-1206 Don`t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natura...

    2018 doi

  13. [21]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, an...

    2022

  14. [22]

    Peng Qian, Tahira Naseem, Roger Levy, and Ram \'o n Fernandez Astudillo. 2021. https://doi.org/10.18653/v1/2021.acl-long.289 Structural guidance for transformer language models . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the...

    2021 doi

  15. [23]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  16. [24]

    Brian Roark. 2001. https://doi.org/10.1162/089120101750300526 Probabilistic top-down parsing and language modeling . Computational Linguistics, 27(2):249--276

    2001 doi

  17. [25]

    Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Milo s Stanojevi \'c , Phil Blunsom, and Chris Dyer. 2022. https://doi.org/10.1162/tacl_a_00526 Transformer grammars: Augmenting transformer language models with syntactic inductive biases at scale . Transactions of the Associ...

    2022 doi

  18. [26]

    Yikang Shen, Shawn Tan, Alessandro Sordoni, Peng Li, Jie Zhou, and Aaron Courville. 2022. https://doi.org/10.18653/v1/2022.acl-long.327 Unsupervised dependency graph network . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2022 doi

  19. [27]

    Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, and Aaron Courville. 2021. https://doi.org/10.18653/v1/2021.acl-long.559 S truct F ormer: Joint unsupervised induction of dependency and constituency structure from masked language modeling . In Proceedings of the 59t...

    2021 doi

  20. [28]

    Mitchell Stern, Daniel Fried, and Dan Klein. 2017. https://doi.org/10.18653/v1/D17-1178 Effective inference for generative neural parsing . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1695--1700, Copenhagen, Denmark. Associa...

    2017 doi

  21. [29]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  22. [30]

    Yaushian Wang, Hung-Yi Lee, and Yun-Nung Chen. 2019. https://doi.org/10.18653/v1/D19-1098 Tree transformer: Integrating tree structures into self-attention . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joi...

    2019 doi

  23. [31]

    Dani Yogatama, Yishu Miao, Gabor Melis, Wang Ling, Adhiguna Kuncoro, Chris Dyer, and Phil Blunsom. 2018. https://openreview.net/forum?id=SkFqf0lAZ Memory architectures in recurrent neural network language models . In International Conference on Learning Representations

    2018

  24. [32]

    Ryo Yoshida and Yohei Oseki. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.428 Composition, attention, or both? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5822--5834, Abu Dhabi, United Arab Emirates. Association for Computational Li...

    2022 doi

  25. [33]

    Yu Zhang, Houquan Zhou, and Zhenghua Li. 2020. https://doi.org/10.24963/ijcai.2020/560 Fast and accurate neural crf constituency parsing . In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 , pages 4046--4053. International J...

    2020 doi

  26. [34]

    Yida Zhao, Chao Lou, and Kewei Tu. 2024. https://doi.org/10.18653/v1/2024.acl-long.84 Dependency transformer grammars: Integrating dependency structures into transformer language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistic...

    2024 doi

  27. [35]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.