Pith. sign in

REVIEW 2 major objections 5 minor 2 cited by

Does BERT agree? Evaluating knowledge of structure dependence through agreement relations

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BERT answers agreement cloze probes correctly in 94.3 percent of examples across 26 languages and four agreement types.

desk verdict A valuable multilingual agreement dataset and an honestly written paper, but the headline 94.3% accuracy rests on a lenient category-level metric that leaves the central structural claim under-supported. read the letter →

arxiv 1908.09892 v1 pith:XC7GUTCM submitted 2019-08-26 cs.CL

Geoff Bacon , Terry Regier This is my paper · ORCID
classification cs.CL
keywords BERTagreementrelationsstructuredependenceclozetestmultilingualmorphosyntacticfeaturesdependencydistancedistractors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the transformer-based language model BERT has internalized structure-dependent agreement rules rather than merely surface statistics. Using a cloze task in which one word of an agreement pair is masked, the authors score BERT against nearly one million naturally occurring examples spanning 26 languages and four agreement relations: subject-verb, determiner-noun, attributive adjective, and predicate adjective. They report that BERT is correct in 94.3% of examples, with accuracy above 90% for every agreement type. Performance slips only slightly but consistently as the distance between the agreeing words grows and as the number of other nouns with conflicting features rises. The result matters because it suggests a model trained without explicit syntax can approximate structure-dependent knowledge in a language-general way, while the distance and distractor effects mark where that approximation fails.

What carries the argument

The load-bearing mechanism is the category-level cloze scoring rule. BERT is a transformer-based neural language model trained to predict masked words; the paper masks the controller noun or pronoun in an agreement relation and asks the model to assign probabilities to vocabulary items. Instead of comparing the masked word's probability with inflectional variants of the same lemma, as earlier work did, the paper averages probabilities across all words matching the masked word's morphosyntactic feature bundle and all words that differ in at least one feature, requiring the former average to exceed the latter. That design, together with a curated corpus of 980,889 naturally occurring agreement examples and feature-bundle annotations harvested from dependency treebanks and morphological lexicons, is what isolates structure-sensitive agreement knowledge from lexical-selection and frequency effects.

What would settle it

Run a frequency-only baseline on the same cloze examples and the same scoring rule; if it approaches 94.3 percent, the grammar-sensitivity claim is falsified. Alternatively, mask both words in the agreement pair and see whether accuracy collapses when no agreeing surface cue remains.

Watch

Extended reading notes

Core claim

The central claim is that BERT encodes syntax-sensitive agreement patterns across languages and agreement types, not just English subject-verb number agreement. On the paper's evaluation, a cloze example counts as correct only when the average probability of all vocabulary words carrying the masked word's feature bundle exceeds the average probability of all words differing in at least one feature; the average example is judged against around 1,468 words. Under that criterion BERT scores 94.3% overall, above 90% in each of the four agreement types, and in all 26 languages above 60% (and above 90% in most). The same analysis shows a small, steady accuracy decline as dependency distance increases and a later but steeper decline as the number of distractor nouns grows, a qualification the authors treat as central to the otherwise strong result.

Load-bearing premise

The 94.3 percent result depends on assuming that beating the averaged-probability comparison proves sensitivity to grammar rather than to word frequency or leftover agreement cues in the sentence.

Editorial extensions

If this is right

  • BERT's agreement sensitivity extends beyond English subject-verb number agreement to determiner, attributive adjective, and predicate adjective relations in 26 languages.
  • Longer dependency distances and more distractor nouns are the identifiable conditions under which this sensitivity degrades, locating the model's structural blind spot.
  • The category-level evaluation criterion offers a reusable way to test whether future pretrained models know agreement relations without being fooled by word frequency.
  • The released dataset of almost one million curated examples gives later work a common benchmark for probing structure dependence across languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the scoring counts any word with the same feature bundle as correct, BERT's 94.3% could partly reflect unmasked agreeing cues elsewhere in the sentence; a dual-masking variant would quantify how much of the score survives when no agreeing surface cue remains.
  • If the distance and distractor declines reflect a real structural limit, then pretraining on longer sentences or with objectives that encourage hierarchical abstraction should improve long-distance agreement accuracy; that is a testable training intervention.
  • The same category-level probing could be applied to other structure-dependent phenomena, such as reflexive binding or case assignment, to see whether the localization of the model's syntactic knowledge is specific to agreement or general.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper evaluates BERT's sensitivity to structure-dependent agreement relations, expanding prior work on English subject-verb number agreement to four agreement types (subject-verb, determiner-noun, attributive adjective, predicate adjective) across 26 languages. The authors build a semi-automatically curated cloze dataset of nearly a million examples from Universal Dependencies treebanks, always masking the controller noun or pronoun, and scoring each example by whether the average probability of all words matching the masked word's feature bundle exceeds that of all words with different features. They report an overall accuracy of 94.3%, with performance decreasing slightly as dependency distance and distractor count increase, and they interpret this as evidence that BERT encodes syntax-sensitive agreement patterns well without explicit syntactic supervision.

Significance. If the central claim holds, this would be a substantial cross-linguistic demonstration that pre-trained transformer models acquire some structure-dependent agreement knowledge. The paper contributes a large new cross-linguistic dataset and code, and it broadens the evaluation to agreement types and features beyond English subject-verb number agreement. The category-level scoring criterion is a genuine attempt to move beyond the top-1 lexical-selection metric of prior work. However, the interpretation of the 94.3% figure as evidence of syntactic sensitivity is not currently secure, because the evaluation procedure may allow surface cues and lexical statistics to drive the result. The load-bearing issue is the measurement strategy, not the data or the experimental effort.

major comments (2)
  1. [Section 3.1 and Section 4] The evaluation criterion does not isolate structure-dependent knowledge because the agreement target is left unmasked. Section 3.1 states that in every cloze example the controller is masked, so the target (the inflected verb, determiner, or adjective) remains visible and directly encodes the feature bundle of the masked word. For instance, in noun-determiner agreement, the determiner's gender and number almost fully determine the noun's features, so a model can assign higher average probability to the correct feature class without representing the syntactic relation. The paper itself acknowledges this redundancy in Section 7 and proposes running the experiment with the target masked, but does not implement that control. Until such a target-masked condition is reported, the abstract's claim that BERT 'captures syntax-sensitive agreement patterns well' is not supported by the current evidence.
  2. [Section 4, no frequency baseline] No frequency-only baseline is provided to interpret the 94.3% accuracy. The category-level criterion compares the average probability of all words sharing the masked word's features with that of all words differing on at least one feature. In many languages, the noun vocabulary is strongly skewed by frequency (e.g., singular forms far more numerous or frequent than plural forms), so a model that ignores context entirely could satisfy the criterion for high-frequency feature classes. A baseline such as predicting the most frequent feature class in the language, or a permuted-word-order control, is necessary to attribute the result to agreement sensitivity rather than corpus statistics. Without such a baseline, the central quantitative claim is ambiguous.
minor comments (5)
  1. [Section 5, Figures 3 and 4] The distance and distractor-count analyses are presented separately, but these factors are likely correlated (longer dependencies often involve more intervening nouns). A joint model or stratified analysis would clarify whether the degradation is attributable to dependency length, distractor count, or both.
  2. [Section 3.2] The paper filters out words that do not appear in BERT's vocabulary, but does not specify how words that are tokenized into multiple WordPiece subword units are handled. Since BERT operates on subword tokens, the treatment of multi-token nouns is important for the validity of the cloze procedure.
  3. [Section 4] The sentence 'The average example in our cloze data is evaluated using 1,468 words' is ambiguous; it is unclear whether this is the average number of candidate word types per masked position or some other quantity. Please clarify the wording.
  4. [Figure 2] The bootstrap confidence intervals are described only as bootstrapped; it would be helpful to state the resampling unit (e.g., cloze examples, sentences, or corpus documents).
  5. [References] The van Schijndel et al. entry is listed as 'forthcoming' without a year or venue; if available, a fuller citation would help readers locate the work.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper evaluates a pretrained model against external annotations, with no fitted parameters or self-citation chain forcing the reported result.

full rationale

The paper's central claim is an empirical evaluation of the pretrained BERT models against externally sourced Universal Dependencies and UniMorph data. No parameter is fitted to the test examples, and the accuracy figures are not constructed from the experimental inputs: BERT's probability assignments are independent of the UD/UniMorph feature labels used to score them. The evaluation metric, which compares the average probability of same-feature words against that of different-feature words, is a direct operationalization of the paper's own definition of agreement as feature covariance, but using that definition to measure BERT's behavior is not circular because the model's outputs are not derived from the feature data. The acknowledged redundancy of unmasked target cues (Section 7, 'in certain languages some of the cloze examples we studied contain redundant information... number information is still available from the determiner') is a validity or control concern about whether high accuracy reflects surface covariation rather than hierarchical structure, not a circularity in the derivation chain. The paper cites prior work for methodology, but none of the load-bearing assumptions reduces to a self-citation or to an unverified uniqueness claim, and no ansatz is imported solely through citation. The result is therefore self-contained as an empirical finding, and no circular step is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities. The central claim depends on the validity of UD/UniMorph annotations, the grammar-based type filtering, and the category-level scoring criterion; these are data and evaluation assumptions rather than fitted quantities.

assumptions (4)
  • domain assumption Universal Dependencies v2.4 annotations reliably identify agreement word pairs and morphosyntactic features across all 26 languages.
    Used at the data curation stage in Section 3.1 to harvest nearly one million cloze examples and to define correct and incorrect words in Section 4. Annotation errors would directly distort accuracy estimates.
  • domain assumption Reference grammars correctly determine which of the four agreement types occur in each language, and the 8% removal of wrong-type examples is accurate.
    Section 3.1: after UD harvesting, examples of agreement types judged absent from a language were deleted. If the grammatical judgments are wrong, the dataset is polluted.
  • ad hoc to paper The category-level criterion (average probability of all feature-matching words versus all non-matching words) measures syntactic sensitivity and controls for selectional restrictions and frequency.
    Section 4 defines correct as average probability of correct words being higher than that of incorrect words. This is not an established benchmark, and without a frequency-only baseline it remains an assumption that this isolates syntax.
  • domain assumption The multilingual BERT model is an appropriate and comparable substitute for single-language models for all non-English languages.
    Section 4: English uses the English BERT, all other languages use multilingual BERT. Cross-language performance differences are therefore not directly comparable, and the paper acknowledges this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Does BERT agree? Evaluating knowledge of structure dependence through agreement relations." pith.science (2026). https://pith.science/paper/XC7GUTCM

@misc{pith2026190809892,
  author       = {Pith},
  title        = {Pith review of: Does BERT agree? Evaluating knowledge of structure dependence through agreement relations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XC7GUTCM}},
  note         = {Machine review of arXiv:1908.09892}
}
read the original abstract

Learning representations that accurately model semantics is an important goal of natural language processing research. Many semantic phenomena depend on syntactic structure. Recent work examines the extent to which state-of-the-art models for pre-training representations, such as BERT, capture such structure-dependent phenomena, but is largely restricted to one phenomenon in English: number agreement between subjects and verbs. We evaluate BERT's sensitivity to four types of structure-dependent agreement relations in a new semi-automatically curated dataset across 26 languages. We show that both the single-language and multilingual BERT models capture syntax-sensitive agreement patterns well in general, but we also highlight the specific linguistic contexts in which their performance degrades.

Figures

Figures reproduced from arXiv: 1908.09892 by the authors.

Figure 1
Figure 1. Accuracy per agreement type aggregated across all languages. In all four types, BERT performed above 90% accuracy. Accuracy is slightly lower for predicate adjectives and subject-verb agreement rela￾tions, which typically have longer distance dependen￾cies. Error bars are bootstrapped 95% confidence inter￾vals [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Accuracy as a function of distance between [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 2
Figure 2. Accuracy per language aggregated across all [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Accuracy as a function of number of distrac [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Quantity doesn't buy quality syntax with neural language models

    cs.CL 2019-08 conditional novelty 6.0 of 10

    More training data and larger LSTM hidden layers yield diminishing returns on subject-verb agreement accuracy, and GPT and BERT sometimes score below LSTMs trained on far less data.

  2. The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

    cs.CL 2026-01 conditional novelty 4.0 of 10

    A systematic review of 337 articles shows Transformers handle formal syntax well but perform worse and more variably at the syntax-semantics interface, with the field over-reliant on English and BERT.

Reference graph

Works this paper leans on

36 extracted references · 18 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mark C. Baker. 2008. https://doi.org/10.1017/CBO9780511619830 The Syntax of Agreement and Concord . Cambridge Studies in Linguistics. Cambridge University Press

    work page doi:10.1017/cbo9780511619830 2008

  4. [4]

    Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49--72

    2019

  5. [5]

    Samuel R Bowman, Christopher D Manning, and Christopher Potts. 2015. Tree-structured composition in neural networks without tree-structured architectures. In Proceedings of the 2015 International Conference on Cognitive Computation: Integrating Neural and Symbolic Approaches-Volume 1583, pages 37--42. CEUR-WS. org

    work page 2015

  6. [6]

    Noam Chomsky. 1965. https://books.google.com/books?id=u0ksbFqagU8C Aspects of the Theory of Syntax . Cambridge, MA. MIT Press

    work page 1965

  7. [7]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does BERT look at? A n analysis of BERT 's attention. arXiv preprint arXiv:1906.04341

    arXiv 2019

  8. [8]

    Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo\" i c Barrault, and Antoine Bordes. 2017. https://www.aclweb.org/anthology/D17-1070 Supervised learning of universal sentence representations from natural language inference data . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670--680, Copenhagen, Denmark. ...

    2017

Show all 36 references
  1. [9]

    Alexis Conneau, Germ \'a n Kruszewski, Guillaume Lample, Lo \"i c Barrault, and Marco Baroni. 2018. http://aclweb.org/anthology/P18-1198 What you can cram into a single vector: Probing sentence embeddings for linguistic properties . In Proceedings of the 56th Annual Meeting of...

    2018

  2. [10]

    Greville G Corbett. 2003. Agreement: terms and boundaries. In The Role of Agreement in Natural Language. Proceedings of the 2001 Texas Linguistic Society Conference, Austin, Texas., pages 109--122

    2003

  3. [11]

    Greville G Corbett. 2006. Agreement, volume 109. Cambridge University Press

    2006

  4. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT : Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  5. [13]

    Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179--211

    1990

  6. [14]

    Allyson Ettinger. 2019. http://arxiv.org/abs/1907.13528 What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models

    2019 arXiv

  7. [15]

    Martin BH Everaert, Marinus AC Huybregts, Noam Chomsky, Robert C Berwick, and Johan J Bolhuis. 2015. Structures, not strings: L inguistics as part of the cognitive sciences. Trends in cognitive sciences, 19(12):729--743

    2015

  8. [16]

    Yoav Goldberg. 2019. Assessing BERT 's syntactic abilities. arXiv preprint arXiv:1901.05287

    2019 arXiv

  9. [17]

    Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. http://aclweb.org/anthology/N18-1108 Colorless green recurrent networks dream hierarchically . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Co...

    2018

  10. [18]

    John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Sho...

    2019

  11. [19]

    Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780

    1997

  12. [20]

    Ganesh Jawahar, Beno \^ t Sagot, Djam \'e Seddah, Samuel Unicomb, Gerardo I \ n iguez, M \'a rton Karsai, Yannick L \'e o, M \'a rton Karsai, Carlos Sarraute, \'E ric Fleury, et al. 2019. What does BERT learn about the structure of language? In 57th Annual Meeting of the Assoc...

    2019

  13. [21]

    Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294--3302

    2015

  14. [22]

    Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside BERT 's linguistic knowledge. arXiv preprint arXiv:1906.01698

    2019 arXiv

  15. [23]

    Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521--535

    2016

  16. [24]

    Rebecca Marvin and Tal Linzen. 2018. https://www.aclweb.org/anthology/D18-1151 Targeted syntactic evaluation of language models . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192--1202, Brussels, Belgium. Association for Com...

    2018

  17. [25]

    Arya D McCarthy, Miikka Silfverberg, Ryan Cotterell, Mans Hulden, and David Yarowsky. 2018. Marrying universal dependencies and universal morphology. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 91--101

    2018

  18. [26]

    R Thomas McCoy, Tal Linzen, and Robert Frank. 2018. Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks. arXiv preprint arXiv:1802.09091

    2018 arXiv

  19. [27]

    Bennett, and Noah D

    Allen Nie, Erin D. Bennett, and Noah D. Goodman. 2017. http://arxiv.org/abs/1710.04334 Dissent: Sentence representation learning from explicit discourse relations . CoRR, abs/1710.04334

    2017 arXiv

  20. [28]

    Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan T McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In LREC

    2016

  21. [29]

    Omer Preminger. 2014. Agreement and its failures. Number 68 in Linguistic Inquiry Monographs. MIT Press, Cambridge, MA

    2014

  22. [30]

    forthcoming

    Marten van Schijndel, Aaron Mueller, and Tal Linzen. forthcoming. Quantity doesn't buy quality syntax with neural language models

  23. [31]

    Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. https://openreview.net/forum?id=B18WgG-CZ Learning general purpose distributed sentence representations via large scale multi-task learning . In International Conference on Learning Representations

    2018

  24. [32]

    John Sylak-Glassman. 2016. The composition and use of the universal morphological feature schema (unimorph schema). Technical report, Technical report, Johns Hopkins University

    2016

  25. [33]

    Wilson L Taylor. 1953. Cloze procedure: A new tool for measuring readability. Journalism Bulletin, 30(4):415--433

    1953

  26. [34]

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. arXiv preprint arXiv:1905.05950

    2019 arXiv

  27. [35]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998--6008

    2017

  28. [36]

    John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. In International Conference on Learning Representations

    2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.