Pith. sign in

REVIEW 2 major objections 8 minor 1 cited by

Sneaking Syntax into Transformer Language Models with Tree Regularization

T0 review · 2 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TreeReg injects tree structure into transformer LMs with an auxiliary orthogonality loss, improving syntactic generalization by up to 9.5 points and out-of-distribution perplexity by up to 9.2% with no architectural change.

desk verdict TreeReg is a genuinely new regularizer for injecting syntax into transformers, but the headline gains are inflated by test-set hyperparameter selection and the abstract oversells the WikiText results. read the letter →

arxiv 2411.18885 v2 pith:RL3OCMED submitted 2024-11-28 cs.CL

Ananjan Nandi , Christopher D. Manning , Shikhar Murty This is my paper · ORCID
classification cs.CL
keywords TreeRegsyntacticinductivebiasorthogonalityconstraintsconstituencyparsingtransformerlanguagemodelsgeneralizationout-of-distributionperplexitySpanContextualIndependenceScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TreeReg is an auxiliary loss that tells a transformer language model which spans of words should behave like independent constituents. It converts bracketings from silver constituency parses into orthogonality constraints on hidden states: a constituent's vector should have zero overlap with the context immediately before and after it. The paper claims this soft syntactic bias improves syntactic generalization by up to 9.5 points on SyntaxGym and lowers out-of-distribution perplexity by up to 9.2%, with the architecture and inference procedure left untouched. If true, this is a practical recipe for adding hierarchical structure to standard transformer LMs, roughly doubling their sample efficiency on syntactic generalization and making fine-tuned models more robust on adversarial NLI benchmarks.

What carries the argument

The Span Contextual Independence Score (SCIN) is the core object. For a span $S_{i;j}$, with $h_k$ the L2-normalized concatenation of hidden states from a chosen subset of attention heads at a chosen layer, it is defined as $\mathrm{SCIN}(i,j) = \|\mathrm{orth}(h_j,h_{i-1})\| + \|\mathrm{orth}(h_{j+1},h_j)\|$, where $\mathrm{orth}(x,y)=x-(x^{\top}y)y$ is the component of $x$ orthogonal to $y$. The first term asks that the span be independent of what precedes it; the second asks that the following token be independent of the span. The TreeReg loss sums, over every constituent in the silver parse, a log-loss that contrasts the score of the silver split with the softmax over all possible splits, thereby raising SCIN on constituents and lowering it on non-constituents. The same greedy read of SCIN scores recovers an induced parse tree at inference without changing the model.

What would settle it

Take a test set of sentences where the most informative word of a phrase is not its last word, such as 'the decision to resign' in 'the decision to resign was announced', decode each sentence with the greedy SCIN parser, and compare against gold parse trees: if the recovered trees systematically miss exactly those phrases while matching elsewhere, the last-token span vector is not carrying the information the orthogonality constraint depends on.

Watch

Extended reading notes

Core claim

The central claim is that the hidden states of an autoregressive transformer can be shaped into a tree-structured computation by a regularizer that is simply added to the language-model loss. For each span in a silver constituency parse, TreeReg drives up the Span Contextual Independence Score (SCIN), defined as the L2 norm of the component of the span's last-token hidden state orthogonal to the hidden state at the preceding boundary, plus the norm of the orthogonal component of the following hidden state relative to the span's; non-constituent spans are pushed in the opposite direction. In an optimally trained model, the tree that maximizes cumulative SCIN over its spans recovers the constituency parse, so the regularizer acts as a soft, differentiable re-encoding of syntax. The authors show that models trained this way from scratch on BLLIP-LG beat an otherwise identical base LM by 2.6 points on BLiMP and 8.1 points on SyntaxGym, lower PTB perplexity by 9.2%, and exceed the baseline's best syntactic generalization with less than half the training data; continued pretraining of Sheared Llama-1.3B with TreeReg improves syntactic generalization, and fine-tuning with TreeReg on MultiNLI reduces the drop on MoNLI from 48.4 points to 6.8 points relative to the unfinetuned model.

Load-bearing premise

The method assumes that the hidden state of the last token of a span faithfully represents that span, so that making that vector orthogonal to its surrounding context really captures the independence of the grammatical phrase.

Editorial extensions

If this is right

  • If TreeReg is correct, an unchanged transformer LM architecture can be given a syntactic inductive bias without additional parameters or inference-time cost.
  • Standard LMs will need less than half the training data to reach the same level of syntactic generalization when TreeReg is added.
  • Out-of-distribution text (PTB) will be modeled with up to 9.2% lower perplexity when TreeReg is used during pretraining.
  • Continued pretraining of an LLM with TreeReg should improve BLiMP, SyntaxGym, and PTB perplexity over continued pretraining without it.
  • Fine-tuning an LLM with TreeReg will preserve much more of its accuracy on adversarial NLI datasets compared with standard fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension beyond the paper is to swap the last-token span vector in Eq. (1) for a mean-pooled or attention-pooled representation; if the gains vanish, the paper's assumption that the final token summarizes the span is doing the work, and if they persist, the mechanism is more general.
  • The paper's greedy parse decoder only sees SCIN on spans that were possible splits of supervised constituents, so the induced tree need not be the global SCIN-maximizing tree; a CKY decoder over all spans would reveal whether the bias is encoding a genuine parse or only a local approximation.
  • Because randomized parses still help BLiMP but not SyntaxGym, some of TreeReg's effect may be a generic regularity push rather than the specific tree shape; comparing TreeReg against an equal-cost regularizer on random spans would isolate the syntactic content.
  • The layerwise F1 curve in Figure 4, rising to layer 12 and collapsing after, is a diagnostic that could be used on any new model to check whether the regularized circuit is actually the one doing syntactic work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. The paper proposes TreeReg, an auxiliary loss that converts constituency bracketing decisions from silver parses into differentiable orthogonality constraints on transformer hidden states. The loss is added to the standard LM objective without architectural changes. The authors report improved syntactic generalization on BLiMP and SyntaxGym, lower out-of-distribution perplexity on PTB, better sample efficiency, and gains in continued pretraining of Sheared Llama and fine-tuning on MultiNLI, including a randomized-parse control that supports the causal role of the parse supervision.

Significance. If the quantitative results hold, this is a useful and novel contribution: it provides a simple, computationally light way to inject hierarchical inductive bias into transformers without modifying the architecture or inference procedure, in contrast to prior approaches that alter attention or add parameters. The randomized-parse experiment is a strong control, and the code release supports reproducibility. The main reservation is that the headline magnitudes are selected on the same test sets used for reporting, so the size of the central gains is not yet established; nevertheless, the method's effectiveness is supported across several independent settings.

major comments (2)
  1. [Appendix G / §5.1, Table 2] The central quantitative claims in Table 2 are based on a configuration (layer 12, 2 of 8 attention heads) that was selected by maximizing SyntaxGym performance and minimizing PTB perplexity on the same test sets used to report those numbers. Because 21 configurations were examined (layers 2,4,6,8,10,12,14 times heads 2,4,8), the reported 8.1-point SG gain and 9.2% PTB improvement are best-of-sweep values and are therefore optimistic. The authors should either choose the configuration on a held-out validation split (e.g., a validation portion of BLLIP-LG or a separate validation benchmark) and then report test metrics for that configuration, or report the distribution of SG and PTB results across all 21 configurations (e.g., mean±std) and base the headline claims on typical or pre-registered performance. This issue is load-bearing because the abstract and Section 5.1's primary quantitative claims rely on the selected numbers.
  2. [§5.3 / Abstract, Table 3] The abstract's claim that "LMs pre-trained with TreeReg on natural language corpora such as WikiText-103 achieve up to 10% lower perplexities on out-of-distribution data" is contradicted by the first block of Table 3, the exact WikiText-103 setting with no LM training on BLLIP-LG: TreeReg increases PTB perplexity from 331.5 to 411.1, a 24% relative degradation. The PTB improvement (2.4 points, about 4.5% relative) appears only when LM training is additionally performed on BLLIP-LG batches, which is a mixed-data setup rather than the advertised "different data sources" scenario. The abstract and Section 5.3 should be qualified to state the condition under which OOD perplexity improves, or the claim should be limited to the same-corpus settings. As written, the general claim overstates the evidence and the failure case is not acknowledged.
minor comments (8)
  1. [Abstract / §6.2, Table 5] The abstract reports "41.2 points" of mitigation on adversarial NLI benchmarks, but the corresponding numbers in Table 5 (MoNLI decrease without TreeReg: 50.3 to 1.9 = 48.4; with TreeReg: 50.3 to 43.5 = 6.8) give a mitigation of 48.4 - 6.8 = 41.6 points. Please correct the value.
  2. [§3.2, Eq. (4)-(5), Alg. 1] The text states that TreeReg "maximizes SCIN for spans corresponding to input constituents while simultaneously minimizing it for other spans," but the loss in Eq. (4) only contrasts alternative splits of gold constituents; spans that are not split candidates of any gold constituent are never directly scored or penalized. Consider rephrasing to "contrasts the gold split against alternative splits of each constituent" or explaining how the contrastive signal also controls non-candidate spans.
  3. [Table 3] The row "No LM on BLLIP-LG" shows a 24% PTB perplexity degradation for TreeReg (411.1 vs 331.5). This failure case should be explicitly discussed in Section 5.3 and reconciled with the abstract's general claim about OOD perplexity improvements, since it directly limits the practical flexibility claim in §3.3.
  4. [Appendix G, Figure 7] Figure 7's axis label reads "PTB Perplexity on SG test suites," which is a copy-paste error; the figure plots PTB perplexity against layer and head count, not SyntaxGym performance.
  5. [Appendix A.1] The justification for using the last token's hidden state as the span representation relies on a citation (Allen-Zhu and Li, 2023) about PCFG-trained transformers. Since the SCIN loss directly supervises these states, an additional diagnostic or representation-level ablation would help substantiate the design choice.
  6. [Tables 2-5] Most results are reported from a single run without standard deviations. Given the hyperparameter sensitivity indicated in Appendix G, reporting variance over seeds for the main configurations would substantially strengthen the reliability of the claimed gains.
  7. [§5.1, Table 2] The comparison with PLM, TG, and Pushdown LM is based on previously published numbers; please state whether those baselines were trained with the same data size, steps, and hyperparameters, or treat the comparison as indicative rather than controlled.
  8. [§5.2 / Conclusion] The statement that "TreeReg more than doubles the sample efficiency of syntactic generalization" is based only on the SyntaxGym benchmark; the conclusion should qualify that the claim refers to that test suite.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline SG/PTB gains in §5.1 are partly a selection artifact because the layer/head configuration was chosen by maximizing the same SyntaxGym and PTB metrics later reported, though the core method is independently supported.

  1. fitted input called prediction [Appendix G and Section 5.1 (Table 2)]
    "we vary the layer used in TREE REG as [2,4,6,8,10,12,14] and the number of attention heads used as [2,4,8] out of 8 ... We find that applying TREE REG at layer 12 on 2 attention heads results in the highest overall SyntaxGym performance as well as lowest PTB perplexity. ... For TREE REG LM, we use LTR at layer 12 of the LM, on 2 out of 8 attention heads"

    Appendix G selects the TreeReg configuration by optimizing the exact metrics later reported as the paper's headline results: the layer/head pair is chosen as the one with the highest overall SyntaxGym performance and lowest PTB perplexity among 21 configurations, and Section 5.1/Table 2 then reports that configuration's SyntaxGym 80.0 and PTB 44.6 as the method's gains. Because the same evaluation sets were used as the selection criterion, the reported SG/PTB numbers are the maximum of the tuning grid rather than independent predictions; the '8.1 pt SG gain' and '9.2% PTB reduction' are partly a selection artifact.

full rationale

TreeReg's training signal (silver parses) and its evaluation metrics (BLiMP, SyntaxGym, PTB, MoNLI, MED) are external to the method's definition, and the SCIN/loss equations (Eqs. 2-5) do not contain the evaluation metrics; there is no self-definitional equivalence. The inspired-by citation to the authors' own tree-projection work (Murty et al. 2023c) is motivational, not load-bearing, and no uniqueness claim is imported. The one legitimate circularity-adjacent issue is in Appendix G: the layer/head configuration used for the main from-scratch experiment was selected by optimizing the exact SyntaxGym and PTB test metrics that Table 2 then reports as headline gains, so those specific numbers are best-of-21 selection values rather than independent predictions. This inflates the claimed magnitude but does not make the central claim circular, because the paper also reports gains in grokking tasks, WikiText settings, continued pre-training, and NLI fine-tuning, plus a randomized-parse ablation showing the signal is syntactic. The parse-recovery analysis is the inverse of the training objective applied to held-out data, not a circular prediction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce new physical entities. It relies on a set of domain assumptions about how to operationalize syntactic context independence in transformer hidden states, plus hyperparameters (layer, heads, frequency, alpha) that are chosen per experiment, with layer and head selection performed on the evaluation benchmarks in Appendix G.

free parameters (4)
  • TreeReg layer = 12 (16-layer), 2 (4-layer), 4 (12-layer GPT-2), 16 (Sheared Llama)
    Layer at which LTR is applied; selected per setting, with Appendix G showing SyntaxGym/PTB performance varies strongly with layer; best layer chosen based on evaluation benchmarks.
  • Number and identity of attention heads = 2/8 (BLLIP-LG scratch), 3/12 (WikiText), 6/24 (Sheared Llama)
    Subset of heads used for SCIN; chosen via ablation in Appendix G.
  • LTR frequency = every 10 LLM steps (most), 20 (warm-up), 5 (continued pre-training)
    How often the auxiliary loss is applied; reported as a hyperparameter without sweep.
  • alpha (regularization weight) = 1
    Set to 1, stated as 'can generally be set to 1'.
assumptions (4)
  • domain assumption Constituent spans should be contextually independent and this property can be enforced by orthogonality constraints.
    Underlies the SCIN definition in §3.1; based on prior work (Murty et al. 2023c).
  • domain assumption The hidden state at the last token of a span summarizes the entire span's content.
    Appendix A.1: 'We use the hidden state for the last token in a span as an easily computable representation of the span'; justified by a citation to Allen-Zhu and Li (2023).
  • domain assumption Silver parses (BLLIP-LG) are accurate enough to serve as supervision.
    Used throughout for LTR targets; no analysis of parse noise effects except the randomized-parse control.
  • domain assumption Standard transformers can encode bounded-depth hierarchical languages without architectural changes (Yao et al., 2021).
    Invoked to justify that soft constraints can work; §8 states 'standard transformers can encode hierarchical languages of bounded depth'.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sneaking Syntax into Transformer Language Models with Tree Regularization." pith.science (2026). https://pith.science/paper/RL3OCMED

@misc{pith2026241118885,
  author       = {Pith},
  title        = {Pith review of: Sneaking Syntax into Transformer Language Models with Tree Regularization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RL3OCMED}},
  note         = {Machine review of arXiv:2411.18885}
}
read the original abstract

While compositional accounts of human language understanding are based on a hierarchical tree-like process, neural models like transformers lack a direct inductive bias for such tree structures. Introducing syntactic inductive biases could unlock more robust and data-efficient learning in transformer language models (LMs), but existing methods for incorporating such structure greatly restrict models, either limiting their expressivity or increasing inference complexity. This work instead aims to softly inject syntactic inductive biases into given transformer circuits, through a structured regularizer. We introduce TreeReg, an auxiliary loss function that converts bracketing decisions from silver parses into a set of differentiable orthogonality constraints on vector hidden states. TreeReg integrates seamlessly with the standard LM objective, requiring no architectural changes. LMs pre-trained with TreeReg on natural language corpora such as WikiText-103 achieve up to 10% lower perplexities on out-of-distribution data and up to 9.5 point improvements in syntactic generalization, requiring less than half the training data to outperform standard LMs. TreeReg still provides gains for pre-trained LLMs: Continued pre-training of Sheared Llama with TreeReg results in improved syntactic generalization, and fine-tuning on MultiNLI with TreeReg mitigates degradation of performance on adversarial NLI benchmarks by 41.2 points. We release all code to guide future research.

Figures

Figures reproduced from arXiv: 2411.18885 by the authors.

Figure 1
Figure 1. TREEREG loss (LTR) computation for S = “he is very happy now”. (i) Computation of vector hidden states hi by passing S as input to some circuit of the LM. hi is the representation for the prefix of S ending at i. (ii) Span Contextual Independence Score (SCIN, § 3.1) computation for “is very happy”. Orthogonality constraints are enforced between span representation h4 and its context h1 and h5. (iii) Chart of SCIN fo… view at source ↗
Figure 2
Figure 2. shows TREEREG improves over the base￾line on 4 out of 6 SG test suites, with a notable 17 pt gain on Licensing. TREEREG LM also generalizes better to PTB with 9.2% lower perplexity, with a marginal 0.7 pt perplexity increase on BLLIP-LG. Licensing Gross Syntactic State Garden Path Effects Center Embedding Agreement Long Distance Dependencies 0 0.2 0.4 0.6 0.8 Base LM TreeReg LM (ours) A c c u r a c y [PITH_FULL_IMA… view at source ↗
Figure 3
Figure 3. Plot of Syntactic Generalization on SG test [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Unlabeled F1 scores on the BLLIP-LG test [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Syntactic Generalization on SG test suites [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Syntactic Generalization on SG test suites for [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: PTB Perplexity on SG test suites for different [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Silver and induced parse for “The company [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 10
Figure 10. Figure 10: Induced parse for “Mr. Helguera is a lawyer, [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Silver and induced parse for “They could [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Silver parse for ‘Four million common shares, via Salomon Brothers Inc.” X X X X Four X million X X common X shares X X , X X via X X Salomon X X X Brothers X Inc X [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 15
Figure 15. Figure 15: Silver parse for “What debts did Qintex group leave?” [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Induced parse for “What debts did Qintex [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Injection of coarse dependency tags into positional embeddings improves syntactic generalization and downstream GLUE performance over no-syntax baselines.

Reference graph

Works this paper leans on

53 extracted references · 45 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Zeyuan Allen-Zhu and Yuanzhi Li. 2023. Physics of language models: Part 1, learning hierarchical language structures. arXiv preprint arXiv:2305.13673

    arXiv 2023

  5. [5]

    Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2024. The reversal curse: LLMs trained on `` A is B '' fail to learn `` B is A ''. In Proceedings of the International Conference on Learning Representations

    work page 2024

  6. [6]

    Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. 2020. On the A bility and L imitations of T ransformers to R ecognize F ormal L anguages. In Proceedings of the Conference on Empirical Methods in Natural Language Processing

    work page 2020

  7. [7]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page 2020

  8. [8]

    David Chiang and Peter Cholak. 2022. Overcoming a theoretical limitation of self-attention. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    work page 2022

Show all 53 references
  1. [9]

    John Cocke. 1969. Programming languages and their compilers: Preliminary notes. New York University

    1969

  2. [10]

    Stephen Crain and Mineharu Nakayama. 1987. Structure dependence in grammar formation. Language, pages 522--543

    1987

  3. [11]

    Ameet Deshpande and Karthik Narasimhan. 2020. Guiding attention for self-supervised learning with transformers. In Findings of the Association for Computational Linguistics: EMNLP

    2020

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies

    2016

  6. [14]

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang (Lorraine) Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. Faith and fate: Limits of trans...

    2023

  7. [15]

    Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-sequence attentional neural machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    2016

  8. [16]

    Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. Syntaxgym: An online platform for targeted evaluation of language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics: System Demonstrations

    2020

  9. [17]

    Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

    2020

  10. [18]

    Yinuo Guo, Zeqi Lin, Jian-Guang Lou, and Dongmei Zhang. 2020. Hierarchical poset decoding for compositional generalization in language. In Proceedings of the International Conference on Neural Information Processing Systems

    2020

  11. [19]

    Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156--171

    2020

  12. [20]

    John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan Brennan. 2018. Finding syntax in human encephalography with beam search. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    2018

  13. [21]

    Hale and Milo s Stanojevi \'c

    John T. Hale and Milo s Stanojevi \'c . 2024. Do LLM s learn a true syntactic universal? In Proceedings of the Conference on Empirical Methods in Natural Language Processing

    2024

  14. [22]

    Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics

    2020

  15. [23]

    Xiang Hu, Pengyu Ji, Qingyang Zhu, Wei Wu, and Kewei Tu. 2024. Generative pretrained structured transformers: Unsupervised syntactic language models at scale. arXiv preprint arXiv:2403.08293

    2024 arXiv

  16. [24]

    Xiang Hu, Haitao Mi, Zujie Wen, Yafang Wang, Yi Su, Jing Zheng, and Gerard de Melo. 2021. R 2 D 2: Recursive transformer based on differentiable tree for interpretable hierarchical language modeling. In Proceedings of the Annual Meeting of the Association for Computational Lin...

    2021

  17. [25]

    John Judge, Aoife Cahill, and Josef van Genabith. 2006. Q uestion B ank: Creating a corpus of parse-annotated questions. In Proceedings of the International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics

    2006

  18. [26]

    Tadao Kasami. 1966. An efficient recognition and syntax-analysis algorithm for context-free languages. Coordinated Science Laboratory Report no. R-257

    1966

  19. [27]

    Nikita Kitaev, Thomas Lu, and Dan Klein. 2022. Learned incremental representations for parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    2022

  20. [28]

    Phong Le and Willem Zuidema. 2015. Compositional distributional semantics with long short term memory. In Proceedings of the Joint Conference on Lexical and Computational Semantics

    2015

  21. [29]

    Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313--330

    1993

  22. [30]

    R Thomas McCoy, Robert Frank, and Tal Linzen. 2020. Does syntax need to grow on trees? sources of hierarchical inductive bias in sequence-to-sequence networks. Transactions of the Association for Computational Linguistics, 8:125--140

    2020

  23. [31]

    Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the Annual Meeting of the Association for Computational Linguistics

    2019

  24. [32]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2022. Pointer sentinel mixture models. In Proceedings of the International Conference on Learning Representations

    2022

  25. [33]

    Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher Manning. 2023 a . Grokking of hierarchical structure in vanilla transformers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

    2023

  26. [34]

    Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher Manning. 2023 b . Pushdown layers: Encoding recursive structure in transformer language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing

    2023

  27. [35]

    Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning. 2023 c . Characterizing intrinsic compositionality in transformers with tree projections. In Proceedings of the International Conference on Learning Representations

    2023

  28. [36]

    Christophe Pallier, Anne-Dominique Devauchelle, and Stanislas Dehaene. 2011. Cortical representation of the constituent structure of sentences. Proceedings of the National Academy of Sciences, 108(6):2522--2527

    2011

  29. [37]

    Peng Qian, Tahira Naseem, Roger Levy, and Ram \'o n Fernandez Astudillo. 2021. Structural guidance for transformer language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language ...

    2021

  30. [38]

    Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Milo s Stanojevi \'c , Phil Blunsom, and Chris Dyer. 2022. Transformer grammars: Augmenting transformer language models with syntactic inductive biases at scale. Transactions of the Association for Computational Linguistics, 1...

    2022

  31. [39]

    Michelle Sheehan, Theresa Biberauer, Ian Roberts, and Anders Holmberg. 2017. The Final-Over-Final Condition: A Syntactic Universal, volume 76. MIT Press

    2017

  32. [40]

    Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In Proceedings of the International Conference on Learning Representations

    2019

  33. [41]

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing

    2013

  34. [42]

    Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing

    2018

  35. [43]

    Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference ...

    2015

  36. [44]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  37. [45]

    Marten van Schijndel, Andy Exley, and William Schuler. 2013. A model of language processing as hierarchic sequential prediction. Topics in cognitive science, 5(3):522--540

    2013

  38. [46]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the International Conference on Neural Information Processing Systems

    2017

  39. [47]

    Yaushian Wang, Hung-Yi Lee, and Yun-Nung Chen. 2019. Tree transformer: Integrating tree structures into self-attention. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing

    2019

  40. [48]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377--392

    2020

  41. [49]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies...

    2018

  42. [50]

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. Sheared llama: Accelerating language model pre-training via structured pruning. In Proceedings of the International Conference on Learning Representations

    2024

  43. [51]

    Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019. Can neural networks understand monotonicity reasoning? In Proceedings of the ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP

    2019

  44. [52]

    Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. 2021. Self-attention networks can process bounded hierarchical languages. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on N...

    2021

  45. [53]

    Daniel H Younger. 1967. Recognition and parsing of context-free languages in time n3. Information and control, 10(2):189--208

    1967

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.