REVIEW 3 cited by
What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about the information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role reversal, albeit with less sensitivity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inferences and role-based event prediction -- and in particular, it shows clear insensitivity to the contextual impacts of negation.
Forward citations
Cited by 3 Pith papers
-
Similarity Gates Approve Reversals: A Validity Audit of Embedding-Cosine Thresholds in Agent Systems
Embedding-cosine thresholds used as meaning gates instead measure lexical overlap, so in the target cases of reversal-versus-paraphrase the gates fire backwards; a matched-pair audit reveals the regime.
-
Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations
DiscoEval is a new benchmark for measuring discourse awareness in sentence embeddings, and Wikipedia-structure training losses modestly change, but do not beat, pretrained encoders.
-
Does BERT agree? Evaluating knowledge of structure dependence through agreement relations
BERT achieves about 94% category-level accuracy on agreement relations across 26 languages and four agreement types, with modest declines as dependency distance and distractor count increase.
Discussion (0). Continue with ORCID to comment.