{"id":"da2f15ac-ca14-4a17-b6cb-d4dd037ffdf6","arxiv_id":"1908.09892","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"BERT achieves about 94% category-level accuracy on agreement relations across 26 languages and four agreement types, with modest declines as dependency distance and distractor count increase.","lead":"This paper tests whether BERT, a widely used language model, respects grammar rules about word agreement across 26 languages and four agreement types. It finds BERT usually does, with modest declines when agreeing words are far apart or distracting nouns sit between them.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 94.3% figure could largely reflect unmasked feature cues and category-level frequency priors; a target-masked condition is needed to separate structural agreement knowledge from surface redundancy.","rationale":"The paper's strongest claim is that BERT captures structure-dependent agreement patterns well in a language-general way, supported by the 94.3% cloze accuracy. The load-bearing assumption is that the Section 4 category-level correctness criterion isolates structural agreement knowledge. The reader's weakest_assumption identifies exactly this evaluation-metric assumption and the absence of a frequency-only baseline, which I agree is the most serious gap. My independent reading strengthens this concern: because Section 3.1 always masks the controller, all four agreement types retain an overt feature-bearing target, and in determiner and attributive-adjective relations the target itself specifies the noun's number/gender. Section 7 explicitly concedes that many cloze examples contain redundant information from unmasked agreeing words. This means the category-average accuracy can in principle be driven by surface feature agreement between the unmasked target and the candidate nouns, rather than by any hierarchical or structure-dependent generalization. The absence of a frequency-only baseline is particularly acute because the correctness rule sums probability mass over all same-feature nouns vs. all different-feature nouns; a language whose vocabularies are skewed toward one feature value (e.g. singular masculine) would yield high accuracy on average even from unigram statistics alone. I do not think this invalidates the paper; the authors transparently flag the redundant-cue issue, release data/code, and use a much more demanding per-example evaluation than Goldberg's two-word comparison. However, the central 94.3% figure and the abstract's unqualified phrasing 'models capture syntax-sensitive agreement patterns well' go beyond what the current protocol can establish. The target-masked and shuffled-control experiments are straightforward with the released code, and the declared limitations make the paper well-suited to a conditional rather than a reject verdict. No concern about fraud or misconduct is raised; the critique is entirely about the gap between the evaluation design and the structural-knowledge claim.","tokens_in":7835,"tokens_out":1949,"duration_ms":17134,"concrete_test":"Re-run the experiment in a target-masked condition: mask the agreeing target (verb, determiner, or adjective) instead of the controller, keeping the controller's feature-marking noun surface cues intact and applying the identical Section 4 category-average correctness rule. If per-type and per-language accuracy remains near the reported 94.3%, the concern is weakened; if accuracy drops substantially in the high-redundancy relation types, the original figure was inflated by unmasked targets. Additionally, run a control in which each sentence's content words are shuffled while preserving the two agreement words adjacent; if accuracy stays near 94.3% under shuffled structure, the result is attributable to lexical co-occurrence or category frequency, not structure dependence.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim, that BERT captures syntax-sensitive agreement patterns well (94.3% correct), rests on a category-level cloze metric that cannot distinguish structure-dependent knowledge from surface cues and lexical statistics. Section 3.1 masks the controller in every example, so the agreeing target remains present and un-masked. In subject-verb agreement, the inflected verb is a strong surface cue to its own number/person; in determiner and attributive adjective relations, the unmasked determiner or adjective jointly determines the controller's features. Section 7 explicitly concedes that many examples contain redundant information (e.g. masking the noun in French attributive adjectival agreement still leaves the number-marked determiner). Under the category-average evaluation of Section 4, a sentence with an unmasked target provides BERT a direct lexical-surface signal for the feature bundle, so the masked noun can be selected by feature co-occurrence without any representation of syntactic structure. The metric also aggregates over all same-feature nouns versus all different-feature nouns, which introduces a strong frequency prior: since rating a language contains many singular nouns, singular-masked examples may pass by global frequency alone, regardless of contextual agreement. No frequency-only baseline, no permutation of word order, and no control with the target masked instead of the controller is reported. The abstract's language-general claim is thus not secured by the reported experiment, although the paper's own Section 7 identifies the redundant-cue limitation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates BERT's sensitivity to structure-dependent agreement relations, expanding prior work on English subject-verb number agreement to four agreement types (subject-verb, determiner-noun, attributive adjective, predicate adjective) across 26 languages. The authors build a semi-automatically curated cloze dataset of nearly a million examples from Universal Dependencies treebanks, always masking the controller noun or pronoun, and scoring each example by whether the average probability of all words matching the masked word's feature bundle exceeds that of all words with different features. They report an overall accuracy of 94.3%, with performance decreasing slightly as dependency distance and distractor count increase, and they interpret this as evidence that BERT encodes syntax-sensitive agreement patterns well without explicit syntactic supervision.","tokens_in":8250,"tokens_out":4095,"duration_ms":42648,"significance":"If the central claim holds, this would be a substantial cross-linguistic demonstration that pre-trained transformer models acquire some structure-dependent agreement knowledge. The paper contributes a large new cross-linguistic dataset and code, and it broadens the evaluation to agreement types and features beyond English subject-verb number agreement. The category-level scoring criterion is a genuine attempt to move beyond the top-1 lexical-selection metric of prior work. However, the interpretation of the 94.3% figure as evidence of syntactic sensitivity is not currently secure, because the evaluation procedure may allow surface cues and lexical statistics to drive the result. The load-bearing issue is the measurement strategy, not the data or the experimental effort.","major_comments":[{"comment":"The evaluation criterion does not isolate structure-dependent knowledge because the agreement target is left unmasked. Section 3.1 states that in every cloze example the controller is masked, so the target (the inflected verb, determiner, or adjective) remains visible and directly encodes the feature bundle of the masked word. For instance, in noun-determiner agreement, the determiner's gender and number almost fully determine the noun's features, so a model can assign higher average probability to the correct feature class without representing the syntactic relation. The paper itself acknowledges this redundancy in Section 7 and proposes running the experiment with the target masked, but does not implement that control. Until such a target-masked condition is reported, the abstract's claim that BERT 'captures syntax-sensitive agreement patterns well' is not supported by the current evidence.","section":"Section 3.1 and Section 4"},{"comment":"No frequency-only baseline is provided to interpret the 94.3% accuracy. The category-level criterion compares the average probability of all words sharing the masked word's features with that of all words differing on at least one feature. In many languages, the noun vocabulary is strongly skewed by frequency (e.g., singular forms far more numerous or frequent than plural forms), so a model that ignores context entirely could satisfy the criterion for high-frequency feature classes. A baseline such as predicting the most frequent feature class in the language, or a permuted-word-order control, is necessary to attribute the result to agreement sensitivity rather than corpus statistics. Without such a baseline, the central quantitative claim is ambiguous.","section":"Section 4, no frequency baseline"}],"minor_comments":[{"comment":"The distance and distractor-count analyses are presented separately, but these factors are likely correlated (longer dependencies often involve more intervening nouns). A joint model or stratified analysis would clarify whether the degradation is attributable to dependency length, distractor count, or both.","section":"Section 5, Figures 3 and 4"},{"comment":"The paper filters out words that do not appear in BERT's vocabulary, but does not specify how words that are tokenized into multiple WordPiece subword units are handled. Since BERT operates on subword tokens, the treatment of multi-token nouns is important for the validity of the cloze procedure.","section":"Section 3.2"},{"comment":"The sentence 'The average example in our cloze data is evaluated using 1,468 words' is ambiguous; it is unclear whether this is the average number of candidate word types per masked position or some other quantity. Please clarify the wording.","section":"Section 4"},{"comment":"The bootstrap confidence intervals are described only as bootstrapped; it would be helpful to state the resampling unit (e.g., cloze examples, sentences, or corpus documents).","section":"Figure 2"},{"comment":"The van Schijndel et al. entry is listed as 'forthcoming' without a year or venue; if available, a fuller citation would help readers locate the work.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The dataset and cross-linguistic scope are valuable, and the authors honestly acknowledge the redundancy limitation in Section 7. However, the central claim that BERT captures syntax-sensitive agreement patterns rests on a metric that can be satisfied by surface cues and frequency priors. The necessary controls (target-masked condition and a frequency baseline) are straightforward to implement and are already hinted at by the authors, so this is appropriately a major revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper gives us the largest multilingual agreement dataset yet: nearly a million cloze examples across 26 languages and four agreement types, with code and data released. That resource is genuinely useful, and the cross-linguistic expansion is a real step beyond the English-only subject-verb work that preceded it. The authors are also transparent: Section 7 explicitly concedes that masking the controller leaves the agreeing target in place, so redundant cues remain, and they name the obvious fix of masking both sides. There are no fitted parameters and the evaluation uses external treebanks and UniMorph, so circularity is not a concern.\n\nThe soft spot is that the central claim—\"BERT captures syntax-sensitive agreement patterns well,\" backed by 94.3% correct—rests on a category-level metric that cannot separate structural knowledge from surface redundancy and lexical frequency. A cloze example counts as correct when the average probability of all feature-matching words exceeds the average of non-matching words, over roughly 1,468 vocabulary items. That is a far more lenient bar than top-word accuracy, and it can pass on global frequency alone. Since the target is never masked, BERT often has a direct surface signal: an inflected verb or determiner already encodes the feature bundle. The paper acknowledges the redundancy but does not quantify how much of the 94.3% it explains. A frequency-only baseline or a target-masked condition would settle that, and the authors list the latter as future work.\n\nI do not think this is fatal. It is a mismatch between the strength of the claim and the strength of the evidence. The dataset, the per-type and per-language breakdowns, and the distance/distractor degradation results are worth having, even if those results are also confounded. The paper would survive review if the conclusions were modest or if the control experiments were added. As it stands, the abstract overstates what the experiment can show.\n\nWho is this for? People working on multilingual probing and evaluation of pretrained models will get value from the dataset and the analysis framework. It deserves a serious referee, not a desk reject, but with the expectation of substantial revision. My recommendation: engage with it, but treat the 94.3% figure as an upper bound on BERT's agreement sensitivity, not proof of syntactic generalization.","headline":"A valuable multilingual agreement dataset and an honestly written paper, but the headline 94.3% accuracy rests on a lenient category-level metric that leaves the central structural claim under-supported.","tokens_in":8597,"tokens_out":1976,"would_cite":true,"duration_ms":21254,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BERT answers agreement cloze probes correctly in 94.3 percent of examples across 26 languages and four agreement types.","keywords":["BERT","agreement relations","structure dependence","cloze test","multilingual","morphosyntactic features","dependency distance","distractors"],"falsifier":"Run a frequency-only baseline on the same cloze examples and the same scoring rule; if it approaches 94.3 percent, the grammar-sensitivity claim is falsified. Alternatively, mask both words in the agreement pair and see whether accuracy collapses when no agreeing surface cue remains.","tokens_in":7644,"feed_emoji":"🌍","tokens_out":10827,"duration_ms":91071,"temperature":0.7,"pith_summary":"This paper asks whether the transformer-based language model BERT has internalized structure-dependent agreement rules rather than merely surface statistics. Using a cloze task in which one word of an agreement pair is masked, the authors score BERT against nearly one million naturally occurring examples spanning 26 languages and four agreement relations: subject-verb, determiner-noun, attributive adjective, and predicate adjective. They report that BERT is correct in 94.3% of examples, with accuracy above 90% for every agreement type. Performance slips only slightly but consistently as the distance between the agreeing words grows and as the number of other nouns with conflicting features rises. The result matters because it suggests a model trained without explicit syntax can approximate structure-dependent knowledge in a language-general way, while the distance and distractor effects mark where that approximation fails.","feed_headline":"BERT agrees 94.3 percent of the time across 26 languages","feed_subtitle":"A masked-word test shows the model tracks grammar in four agreement types, slipping on long-distance cases.","key_machinery":"The load-bearing mechanism is the category-level cloze scoring rule. BERT is a transformer-based neural language model trained to predict masked words; the paper masks the controller noun or pronoun in an agreement relation and asks the model to assign probabilities to vocabulary items. Instead of comparing the masked word's probability with inflectional variants of the same lemma, as earlier work did, the paper averages probabilities across all words matching the masked word's morphosyntactic feature bundle and all words that differ in at least one feature, requiring the former average to exceed the latter. That design, together with a curated corpus of 980,889 naturally occurring agreement examples and feature-bundle annotations harvested from dependency treebanks and morphological lexicons, is what isolates structure-sensitive agreement knowledge from lexical-selection and frequency effects.","core_discovery":"The central claim is that BERT encodes syntax-sensitive agreement patterns across languages and agreement types, not just English subject-verb number agreement. On the paper's evaluation, a cloze example counts as correct only when the average probability of all vocabulary words carrying the masked word's feature bundle exceeds the average probability of all words differing in at least one feature; the average example is judged against around 1,468 words. Under that criterion BERT scores 94.3% overall, above 90% in each of the four agreement types, and in all 26 languages above 60% (and above 90% in most). The same analysis shows a small, steady accuracy decline as dependency distance increases and a later but steeper decline as the number of distractor nouns grows, a qualification the authors treat as central to the otherwise strong result.","pith_inferences":["Because the scoring counts any word with the same feature bundle as correct, BERT's 94.3% could partly reflect unmasked agreeing cues elsewhere in the sentence; a dual-masking variant would quantify how much of the score survives when no agreeing surface cue remains.","If the distance and distractor declines reflect a real structural limit, then pretraining on longer sentences or with objectives that encourage hierarchical abstraction should improve long-distance agreement accuracy; that is a testable training intervention.","The same category-level probing could be applied to other structure-dependent phenomena, such as reflexive binding or case assignment, to see whether the localization of the model's syntactic knowledge is specific to agreement or general."],"forward_implications":["BERT's agreement sensitivity extends beyond English subject-verb number agreement to determiner, attributive adjective, and predicate adjective relations in 26 languages.","Longer dependency distances and more distractor nouns are the identifiable conditions under which this sensitivity degrades, locating the model's structural blind spot.","The category-level evaluation criterion offers a reusable way to test whether future pretrained models know agreement relations without being fooled by word frequency.","The released dataset of almost one million curated examples gives later work a common benchmark for probing structure dependence across languages."],"supporting_citations":[{"why":"introduces the BERT architecture and its masked-language-model training objective that the cloze probes rely on.","marker":"Devlin et al. (2018)"},{"why":"supplies the original cloze-test setup for BERT and the English subject-verb baseline this paper extends.","marker":"Goldberg (2019)"},{"why":"established the agreement-accuracy methodology for testing neural language models on syntax-sensitive dependencies.","marker":"Linzen et al. (2016)"},{"why":"motivates the category-level probability comparison by showing that word-level agreement success can be contaminated by lexical and frequency effects.","marker":"Gulordava et al. (2018)"},{"why":"provides the argument for using naturally occurring sentences rather than minimal-pair templates in targeted syntactic evaluation.","marker":"Marvin and Linzen (2018)"},{"why":"supplies the multilingual dependency treebank data from which the paper's nearly one million cloze examples are harvested.","marker":"Nivre et al. (2016)"},{"why":"supplies the cross-linguistic feature-bundle schema used to determine which vocabulary words count as agreeing with the masked controller.","marker":"Sylak-Glassman (2016)"},{"why":"defines the controller-target agreement terminology and the typology of agreement relations the study operationalizes.","marker":"Corbett (2006)"},{"why":"supports the four cross-linguistically common agreement types and the morphosyntactic features used in the dataset.","marker":"Baker (2008)"}],"fun_headline_variants":["BERT tracks agreement across 26 languages, but long-distance syntax trips it up","Syntax-smart BERT: 94.3% agreement accuracy across 26 languages","BERT's grammar knowledge spans 26 languages, but distance hurts","BERT gets agreement right 94.3% of the time, but long-distance syntax trips it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 94.3 percent result depends on assuming that beating the averaged-probability comparison proves sensitivity to grammar rather than to word frequency or leftover agreement cues in the sentence.","fun_headline_variants_meta":{"raw":{"variants":["BERT tracks agreement across 26 languages, but long-distance syntax trips it up","Syntax-smart BERT: 94.3% agreement accuracy across 26 languages","BERT's grammar knowledge spans 26 languages, but distance hurts","BERT gets agreement right 94.3% of the time, but long-distance syntax trips it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2789,"prompt_tokens":814,"completion_tokens":1975,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":1889}},"tokens_in":430,"tokens_out":1975,"duration_ms":14284,"temperature":1.0,"reasoning_tokens":1889,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:57:49.983612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a frequency-only baseline on the same cloze examples and the same scoring rule; if it approaches 94.3 percent, the grammar-sensitivity claim is falsified. Alternatively, mask both words in the agreement pair and see whether accuracy collapses when no agreeing surface cue remains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"established the agreement-accuracy methodology for testing neural language models on syntax-sensitive dependencies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"motivates the category-level probability comparison by showing that word-level agreement success can be contaminated by lexical and frequency effects."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the argument for using naturally occurring sentences rather than minimal-pair templates in targeted syntactic evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the cross-linguistic feature-bundle schema used to determine which vocabulary words count as agreeing with the masked controller."}],"review_version":1}