Pith. sign in

REVIEW 2 major objections 6 minor 42 references

MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference

T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read NLI models lose 4–9 percent on minimal word swaps

desk verdict A useful, mostly sound NLI robustness study whose headline gap is plausible but whose label-inheritance assumption needs fuller validation or qualification before the numbers are load-bearing. read the letter →

arxiv 2510.24295 v2 pith:OILXJJUF submitted 2025-10-28 cs.CL

Mădălina Zgreabăn , Tejaswini Deoskar , Lasha Abzianidze This is my paper · ORCID
classification cs.CL
keywords naturallanguageinferencegeneralizationtestingminimaleditscontrastsetsmaskedmodelsrobustnesspatternaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that state-of-the-art natural language inference (NLI) models are not robust to even the mildest, reasoning-preserving surface changes. The authors propose MERGE, a test that creates many variants of each original NLI problem by replacing open-class words shared by the premise and hypothesis with more probable, same-part-of-speech substitutes suggested by masked language models. Because the variants keep the original reasoning, word overlap, and syntax, any drop in accuracy is attributed to a real generalization failure rather than to a shift in test conditions. Across fourteen models, they report that at a 90 percent variant-quality threshold, sample accuracy falls by about 4–9 percent relative to seed problems, and reaching seed-level pattern accuracy requires lowering the accuracy threshold to roughly 60 percent for most models.

What carries the argument

The central mechanism is the MERE generation pipeline: for each seed problem, every open-class word shared between premise and hypothesis is masked in both sentences, and a variant word is kept only if it is suggested by at least one masked language model at every occurrence of that word in both premise and hypothesis, is more probable than the original word under that model, shares the same part of speech, and does not occur elsewhere in the problem. This intersection rule is what guarantees that the replacement is equally felicitous in premise and hypothesis, thereby preserving the seed's reasoning and word overlap. Pattern accuracy over at least 20 variants per seed is then the evaluation device that turns individual predictions into a consistency measure, and thresholds such as 90 percent define what counts as robust.

What would settle it

Re-annotate a random sample of, say, 500 MERGE variants with fresh annotators who see no seed label, and compare their labels to the inherited seed labels; if more than about 10 percent of variants receive a different label, the reported 4–9 percent accuracy drop is substantially contaminated by label-inheritance errors. A complementary check is to recompute the pattern-accuracy curves after dropping variants whose seed problems the paper itself identifies as unreliable.

Watch

Extended reading notes

Core claim

The central discovery is that NLI models generalize poorly under the simplest possible generalization test: replacing a shared open-class word in the premise and hypothesis with a more probable, same-part-of-speech word that preserves the original logical relation. The paper builds MERGE by masking each shared word in both premise and hypothesis, collecting suggestions from several masked language models, and keeping only suggestions that are more probable than the original word, belong to the same part of speech, and are validated at every occurrence of the word in both sentences. The resulting variants are scored with pattern accuracy, which credits a seed problem only when a model correctly classifies at least an x fraction of its variants. At the chosen 90 percent threshold, models are 4–9 percent less accurate on variants than on the original seed problems, and most models need a 60 percent accuracy threshold to match their seed-level pattern accuracy. The authors also find that noun replacements are harder than verb replacements, that the MLM used to generate variants does not systematically favor its NLI counterpart, and that the sheer number of unique variants affects scores more than stricter filtering for plausibility.

Load-bearing premise

The load-bearing premise is that every variant produced by replacing a shared word with a more probable, same-part-of-speech word inherits the seed problem's entailment label; the paper manually spot-checks only a few hundred examples and notes in a footnote that incorrect label inheritance is possible.

Editorial extensions

If this is right

  • Standard in-distribution NLI accuracy systematically overstates reasoning ability: models scoring above 90 percent on SNLI fall 4–9 percent on these minimally altered, reasoning-preserving variants.
  • Pattern accuracy at a threshold is a more informative evaluation than sample-level accuracy, because it reveals when a model fails whole problem families even while getting most individual variants right.
  • NLI models lean on surface form, since swapping words that are irrelevant to the inference, such as changing 'small' to 'little' in 'A small girl carries a girl', produces measurable instability.
  • Robustness-test design should control the number of unique variants, because lexical diversity affects model scores more than stricter plausibility filters.
  • The MERGE methodology is model- and dataset-agnostic, so it can be applied to other NLI benchmarks and to other natural-language tasks that can be framed as NLI.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If MERGE's reasoning-preservation claim is right, the 4–9 percent drop is a lower bound on fragility: adversarial or distribution-shifting tests would likely show larger gaps, so robustness research should use MERGE as a baseline rather than a worst case.
  • Because the variant space is defined by what the selected masked language models find plausible, the absolute accuracy numbers are partly a property of the generation pipeline; a different MLM set could change the difficulty ranking of the same models.
  • A direct test of the paper's token-frequency observation would correlate per-variant accuracy with the frequency of the replacement word in the model's training corpus; the paper's Section 5.5 hints that rare replacements drive part of the drop for SNLI-only models.
  • Fine-tuning on MERGE variants (data augmentation) is an untested extension; with roughly 190k unique variants the experiment is feasible and would show whether the observed fragility can be trained away.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces MERGE, a methodology for constructing minimally altered NLI problems by replacing open-class words shared between premise and hypothesis with higher-probability, same-POS suggestions from a panel of MLMs. Variants inherit the seed label, and the filtering is designed to preserve word overlap and syntax. The authors evaluate 14 NLI models on variants derived from SNLI test problems, using both sample accuracy and pattern accuracy (a per-seed threshold-consistency metric). They report that at a 90% quality threshold, pattern accuracy drops by roughly 4-9 percentage points relative to seed accuracy, and that matching seed-level pattern accuracy requires lowering the threshold to about 60% for most models. Additional analyses examine the effect of replacement POS class, the origin MLM, and different filtering criteria.

Significance. If the label-preservation assumption is adequately validated, MERGE is a practical and low-cost generalization test: it preserves word overlap and syntax by construction, and it explicitly checks a common confound in automatic contrast-set generation, namely whether NLI models are favored when variants come from their own MLM family (Section 5.3). The empirical result that strong NLI models are not robust to minimal, label-preserving lexical substitutions would be a useful data point for the robustness literature. The paper is also candid about limitations, including the possibility of label-changing variants (footnote 4) and unreliable seed labels in SNLI (Section 5.5). A notable weakness is that the final variant dataset is not released, which limits independent verification of the central label-preservation claim.

major comments (2)
  1. [Section 3, Eq. (1); Section 4, Variants Manual Annotation] The entire evaluation treats every variant <P_ij, H_ij, l> as inheriting the seed label l. The only direct validation is a spot-check of 300 variants (100 per POS class) plus 100 post-BART variants, annotated by two authors with no inter-annotator agreement reported, and footnote 4 concedes that label-changing variants are still possible. Because pattern accuracy at QT=90% with 20 variants per seed allows at most two errors before a seed fails, a label-preservation failure rate of even a few percent can push a non-trivial fraction of seeds over the threshold and directly inflate the reported QT and MT values. Section 5.5 shows that seed labels themselves are noisy (only 29% of the 31 hardest seeds have a correct gold label), so the assumption cannot be secured by appealing to seed quality. I therefore ask for a larger random human-validation sample with agreement statistics, or a release of the final variants so that label preservation can be independently audited.
  2. [Section 5.1, Table 3] The headline 4-9% drop is computed as SA on ALLSeed minus PA on ALLVar at the 90% threshold, which mixes a per-example metric with a per-seed threshold metric. The per-example accuracy difference between ALLSeed and ALLVar is much smaller in Table 3 (e.g., 89.6 vs. 88.9 for BERT-B-S and 90.5 vs. 89.1 for OPT-1.3B). I do not dispute that pattern accuracy is a useful consistency metric, but the paper should state explicitly that the 4-9% figure is a pattern-accuracy drop rather than a sample-accuracy drop, and the abstract's wording 'perform 4-20% worse on variants' should be revised to name the metric.
minor comments (6)
  1. [Table 3] The table is hard to parse because adjacent values are concatenated without separators (e.g., '88.9-4.959'); please format the columns properly.
  2. [Abstract] The abstract's '4-20%' range should specify that the lower bound is the pattern-accuracy drop at QT and the upper bound occurs at stricter thresholds; the body reports 4-9% at QT.
  3. [Section 5.4] The comparison between filtering criteria is confounded by dataset size: P_union_H, POS, and None have about 380k unique variants each while ALLVar has about 190k. The conclusion that 'the number of variants seems more important' should be supported by a size-matched subsample.
  4. [Section 5.3] The definition of Multi-MLM as 'any two MLMs, potentially including the evaluated model' is ambiguous; clarify whether the evaluated model's own MLM can be one of the two, since this affects the interpretation of the Multi-MLM curve.
  5. [Section 5.5] The re-annotation of the 31 hard seeds is done by the authors and used to argue that seed labels are unreliable; reporting annotation agreement or a second annotator would strengthen this point.
  6. [Section 4] Please provide a reproducibility statement with code and dataset release information; the current version does not state whether the variant dataset will be made available.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported generalization drop is an empirical measurement against external labels, not a constructional identity.

full rationale

The paper's derivation chain is not circular. MERGE constructs variants by replacing shared open-class words with same-POS, higher-probability MLM suggestions; variant labels are inherited from the seed by Eq. 1, and model predictions are compared with those labels to compute sample and pattern accuracy. No parameter is fitted to model outputs and then reported as a prediction, and the headline 4-9% QT drop and the MT about 60% thresholds are empirical aggregates, not identities forced by the construction. The only self-citation is the pattern-accuracy metric from Abzianidze et al. (2023), a co-authored prior paper; PA is a transparent definition and is not used to derive the drop, so this citation is not load-bearing. The possible circularity-adjacent bias, evaluated NLI models sharing a pretrained backbone with the generating MLMs, is explicitly tested in Section 5.3 and not confirmed. The weakest premise is label inheritance: footnote 4 concedes that variants with incorrect inference labels are still possible, and the manual check covers only 100 variants per POS class plus 100 post-BART examples (91% good), so a residual label-preservation failure could inflate the reported drop. Section 5.5 further shows seed labels themselves are sometimes wrong. These are benchmark-validity risks, not circular reductions: a model prediction is not equal to the inherited label by construction, and the drop is measured, not derived. The derivation chain is self-contained against external SNLI labels and model predictions, so no significant circularity is present.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on design choices and domain assumptions rather than on fitted parameters. The variant space is defined by hand-set thresholds (suggestion count, inflation degree, quality threshold, good-variant cutoff), and the key assumption that label-preserving, same-POS, high-probability replacements exist at scale is only partially verified. No new entities are introduced.

free parameters (6)
  • suggestion_count_per_masked_position = 200
    Each MLM returns 200 candidate words per masked position; a practical ceiling that bounds the variant pool.
  • degree_of_inflation_d = 20
    Number of variants sampled per seed problem per class; chosen for interpretability and to balance variants across seeds.
  • minimum_suggestions_per_seed = 20
    Seed problems with fewer than 20 filtered suggestions are excluded; this hand-set inclusion criterion shapes the final seed set.
  • quality_threshold_QT = 90 percent
    The pattern-accuracy threshold used for the headline 4 to 9 percent drop; chosen partly because 91 percent of a validation sample was rated good.
  • good_variant_threshold_F+R = 9 on a scale of 2 to 10 (1-5 fluency plus 1-5 reasoning)
    Manual annotation cutoff for calling a variant good; used to exclude BART and to estimate dataset quality.
  • probability_filter_relation = suggestion probability greater than original word probability
    The requirement that replacements be more probable than the original word under the MLM; a modeling choice defining 'felicitous'.
assumptions (6)
  • domain assumption Replacing shared open-class words with same-POS, higher-probability MLM suggestions preserves the NLI label for the large majority of variants.
    Invoked in Section 3 when the seed label is inherited by every variant; only spot-checked manually, and the authors acknowledge exceptions in footnote 4.
  • domain assumption MLM probability is a valid proxy for semantic plausibility and label preservation.
    Used in Section 3 constraint (i) to require suggestions more probable than the original word; no independent proof that high MLM likelihood correlates with preserved inference.
  • domain assumption Manual annotation of a few hundred sampled variants generalizes to the full variant dataset.
    Used in Section 4 and Figure 10 to claim 91 percent of variants are good; the full set is not annotated.
  • domain assumption Preserving word overlap, syntax, and sentence length controls for the known NLI heuristics, so score drops reflect reasoning failures.
    This is the design rationale in Sections 1 and 2; it is plausible but not directly verified that these are the only shortcuts models use.
  • domain assumption SNLI gold labels are reliable enough to inherit as variant labels.
    Section 5.5 shows that some of the hardest seed problems may have incorrect gold labels, which implies some inherited variant labels are wrong.
  • standard math Standard paired t-tests are appropriate for comparing pattern-accuracy scores across thresholds.
    Used in Section 5 to report significance; multiple comparisons are not corrected.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference." pith.science (2026). https://pith.science/paper/OILXJJUF

@misc{pith2026251024295,
  author       = {Pith},
  title        = {Pith review of: MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OILXJJUF}},
  note         = {Machine review of arXiv:2510.24295}
}
read the original abstract

As many benchmarks have become saturated, it has become increasingly important to create new datasets that evaluate the generalization capacity of current state-of-the-art models in reasoning. However, designing high-quality reasoning datasets is challenging, as their manual construction is costly, and their automatic generation is unreliable, often leading to synthetic data with limited scope. In this paper, we propose the Minimal Expression-Replacement GEneralization (MERGE) test that evaluates the robustness of reasoning models against non-adversarial variants of existing evaluation datasets. We automatically obtain high-quality variants from the original instances with Minimal Expression REplacement (MERE) generation, which uses Masked Language Models (MLMs) and safeguarding filters. We apply the MERGE test to Natural Language Inference (NLI), a popular task of reasoning. We generate new NLI datasets from two widely used existing ones with the MERE generation and use them to evaluate multiple strong NLI models. The results indicate that both LLMs and fine-tuned NLI models generalize poorly: they struggle to consistently and correctly classify variants minimally different in form and reasoning from the original ones. Further, we also analyze how certain aspects in variant generation, such as the word class and the source MLMs, affect model performance.

Figures

Figures reproduced from arXiv: 2510.24295 by the authors.

Figure 1
Figure 1. MERGE vs. standard sample-based evalua￾tions: while in the former each variant is an independent example, in MERGE performance is measured as the proportion of correctly classified variants for an NLI seed problem, i.e. whether a model classifies at least x amount of variants (the threshold number) for each NLI problem. shown models’ poor generalization more effec￾tively than traditional in-distribution ones such as… view at source ↗
Figure 2
Figure 2. Generating NLI problem variants with MLMs. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Averaged Fluency and Reasoning scores with normalized counts for 100 random variants for Nouns, Verbs, and Adjectives. The red lines are the bar plots weighted considering the distribution of classes in the seed NLI problems (N=67%, V=23%, ADJ=10%). Good variants have a score of F + R >= 9. fewer than 205 vi suggestions (d = 20) after fil￾tering, from all their summed w1, meaning each remaining seed problem had to y… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: PA scores of models on ALLVar from 80% threshold on. The red dots are PA scores at QT of ALLVar (90%). with previous studies (Li et al., 2020; Srikanth and Rudinger, 2025), both RoBERTa NLI mod￾els significantly outperform BERT, though only RoBERTa-L-SMFA surpasses bot…
Figure 5
Figure 5. Figure 5: Averaged PA scores on NVar, VVar, and AVar. 5.2 Which word classes are more difficult? We test models on the ALLVar and its classes from [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Averaged PA curves of all models on variants [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Averaged PA curves of all models on datasets [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 7
Figure 7. Figure 7: Averaged PA curves for BERT, RoBERTa, Electra and ALBERT, all sizes tested, on variants di￾vided by their origin MLM. w c – POS; iii) vij only having wi> – Prob; iv) vij of any class or probability – None; v) vij with their letters randomly scrambled – Scr. Except for …
Figure 9
Figure 9. Figure 9: Confusion matrix showing how models’ sug [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 11
Figure 11. Figure 11: PA scores from threshold 0% for all models on ALL [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 10 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Lasha Abzianidze, Joost Zwarts, and Yoad Winter. 2023. https://aclanthology.org/2023.naloma-1.2/ S pace NLI : Evaluating the consistency of predicting inferences in space . In Proceedings of the 4th Natural Logic Meets Machine Learning Workshop, pages 12--24, Nancy, France. Association for Computational Linguistics

    work page 2023

  4. [4]

    Erik Arakelyan, Zhaoqi Liu, and Isabelle Augenstein. 2024. https://doi.org/10.18653/v1/2024.eacl-long.27 Semantic sensitivities and inconsistent predictions: Measuring the fragility of NLI models . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432--444, St. Jul...

    doi:10.18653/v1/2024.eacl-long.27 2024

  5. [5]

    Jean-Philippe Bernardy and Stergios Chatzikyriakidis. 2019. https://www.scitepress.org/PublishedPapers/2019/76835/76835.pdf What kind of natural language inference are nlp systems learning: Is this enough? In ICAART (2), pages 919--931

    work page 2019

  6. [6]

    Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. 2020. https://arxiv.org/abs/1908.05739 Abductive commonsense reasoning . Preprint, arXiv:1908.05739

    arXiv 2020

  7. [7]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. https://arxiv.org/abs/1508.05326 A large annotated corpus for learning natural language inference . Preprint, arXiv:1508.05326

    arXiv 2015

  8. [8]

    Mikhail Budnikov, Anna Bykova, and Ivan P Yamshchikov. 2025. https://link.springer.com/content/pdf/10.1007/s00521-024-10827-6.pdf Generalization potential of large language models . Neural Computing and Applications, 37(4):1973--1997

    work page doi:10.1007/s00521-024-10827-6.pdf 2025

Show all 42 references
  1. [9]

    Le, and Christopher D

    Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. https://arxiv.org/abs/2003.10555 Electra: Pre-training text encoders as discriminators rather than generators . Preprint, arXiv:2003.10555

    2020 arXiv

  2. [10]

    Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. https://d1wqtxts1xzle7.cloudfront.net/40355261/The_PASCAL_Recognising_Textual_Entailmen20151124-18518-fllk05-libre.pdf?1448469095=&response-content-disposition=inline In Machine learning challenges workshop, pages 177--190....

    2005

  3. [11]

    Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. https://api.semanticscholar.org/CorpusID:52182179 Transforming question answering datasets into natural language inference datasets . ArXiv, abs/1809.02922

    2018 arXiv

  4. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    2019 arXiv

  5. [14]

    Ritam Dutt, Sagnik Ray Choudhury, Varun Venkat Rao, Carolyn Rose, and V.G.Vinod Vydiswaran. 2024. https://doi.org/10.18653/v1/2024.genbench-1.11 Investigating the generalizability of pretrained language models across multiple dimensions: A case study of NLI and MRC . In Procee...

    2024 doi

  6. [15]

    Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A

    Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang ...

    2020 arXiv

  7. [16]

    Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. https://doi.org/10.18653/v1/P18-2103 Breaking NLI systems with sentences that require simple lexical inferences . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short P...

    2018 doi

  8. [17]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://arxiv.org/abs/2006.03654 Deberta: Decoding-enhanced bert with disentangled attention . Preprint, arXiv:2006.03654

    2021 arXiv

  9. [18]

    Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, et al. 2023. https://www.nature.com/articles/s42256-023-00729-y A taxonomy and review of generalization res...

    2023

  10. [19]

    Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. 2020. https://arxiv.org/abs/1909.12434 Learning the difference that makes a difference with counterfactually-augmented data . Preprint, arXiv:1909.12434

    2020 arXiv

  11. [20]

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. https://arxiv.org/abs/1909.11942 ALBERT: A lite BERT for self-supervised learning of language representations . CoRR, abs/1909.11942

    2019 arXiv

  12. [21]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. https://arxiv.org/abs/1910.13461 Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehensi...

    2019 arXiv

  13. [22]

    Chuanrong Li, Lin Shengshuo, Zeyu Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkeld. 2020. https://doi.org/10.18653/v1/2020.blackboxnlp-1.12 Linguistically-informed transformations ( LIT ): A method for automatically generating contrast sets . In Proceedings of the Third...

    2020 doi

  14. [23]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  15. [24]

    Lovish Madaan, David Esiobu, Pontus Stenetorp, Barbara Plank, and Dieuwke Hupkes. 2024. https://arxiv.org/abs/2411.14103 Lost in inference: Rediscovering the role of natural language inference for large language models . Preprint, arXiv:2411.14103

    2024 arXiv

  16. [25]

    Thomas McCoy, Ellie Pavlick, and Tal Linzen

    R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. https://doi.org/10.18653/v1/P19-1334 Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p...

    2019 doi

  17. [26]

    Pablo Miralles-González, Javier Huertas-Tato, Alejandro Martín, and David Camacho. 2025. https://arxiv.org/abs/2504.18376 Pushing the boundary on natural language inference . Preprint, arXiv:2504.18376

    2025 arXiv

  18. [27]

    Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. https://aclanthology.org/C18-1198/ Stress test evaluation for natural language inference . In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340--...

    2018

  19. [28]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. https://arxiv.org/pdf/1910.14599 Adversarial nli: A new benchmark for natural language understanding . arXiv preprint arXiv:1910.14599

    2019 arXiv

  20. [29]

    Ellie Pavlick and Tom Kwiatkowski. 2019. https://doi.org/10.1162/tacl_a_00293 Inherent disagreements in human textual inferences . Transactions of the Association for Computational Linguistics, 7:677--694

    2019 doi

  21. [30]

    Daniel Petrov. 2025. https://arxiv.org/abs/2501.02683 From superficial patterns to semantic understanding: Fine-tuning language models on contrast sets . arXiv preprint arXiv:2501.02683

    2025 arXiv

  22. [31]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://storage.prod.researchhub.com/uploads/papers/2020/06/01/language-models.pdf Language models are unsupervised multitask learners

    2019

  23. [32]

    Sara Rajaee, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.725 Looking at the overlooked: An analysis on the word-overlap bias in natural language inference . In Proceedings of the 2022 Conference on Empirical Methods in ...

    2022 doi

  24. [33]

    Logan IV, Matt Gardner, and Sameer Singh

    Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. 2022. https://arxiv.org/abs/2202.07206 Impact of pretraining term frequencies on few-shot reasoning . Preprint, arXiv:2202.07206

    2022 arXiv

  25. [34]

    Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A

    Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.418 Thinking like a skeptic: Defeasible inference in natural language . In Findings of the As...

    2020 doi

  26. [35]

    Neha Srikanth, Marine Carpuat, and Rachel Rudinger. 2024. https://doi.org/10.1162/tacl_a_00692 How often are errors in natural language reasoning due to paraphrastic variability? Transactions of the Association for Computational Linguistics, 12:1143--1162

    2024 doi

  27. [36]

    Neha Srikanth and Rachel Rudinger. 2025. https://arxiv.org/abs/2502.08080 Nli under the microscope: What atomic hypothesis decomposition reveals . Preprint, arXiv:2502.08080

    2025 arXiv

  28. [37]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. https://doi.org/10.18653/v1/N18-1074 FEVER : a large-scale dataset for fact extraction and VER ification . In Proceedings of the 2018 Conference of the North A merican Chapter of the Associatio...

    2018 doi

  29. [38]

    Dhruv Verma, Yash Kumar Lal, Shreyashee Sinha, Benjamin Van Durme, and Adam Poliak. 2023. https://doi.org/10.18653/v1/2023.acl-short.76 Evaluating paraphrastic robustness in textual entailment models . In Proceedings of the 61st Annual Meeting of the Association for Computatio...

    2023 doi

  30. [39]

    Leon Weber-Genzel, Siyao Peng, Marie-Catherine de Marneffe, and Barbara Plank. 2024. https://arxiv.org/abs/2403.01931 Varierr nli: Separating annotation error from human label variation . Preprint, arXiv:2403.01931

    2024 arXiv

  31. [40]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

  32. [41]

    Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, and Yue Zhang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.276 Out-of-distribution generalization in natural language processing: Past, present, and...

    2023 doi

  33. [42]

    Carbonell, Ruslan Salakhutdinov, and Quoc V

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. https://arxiv.org/abs/1906.08237 Xlnet: Generalized autoregressive pretraining for language understanding . CoRR, abs/1906.08237

    2019 arXiv

  34. [43]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2...

    2022 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.