From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets

Daniel Petrov

REVIEW 3 major objections 5 minor 7 references

From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a pre-trained NLI model on a small subset of contrast examples lifts its accuracy on held-out contrast data from 74.9% to 90.7% while leaving standard test accuracy almost unchanged.

desk verdict Contrast-set fine-tuning recovers accuracy on the same contrast-set distribution, but the 'semantic understanding' framing doesn't survive contact with the evaluation design. read the letter →

arxiv 2501.02683 v2 pith:LYZJZUUH submitted 2025-01-05 cs.CL cs.AI

Daniel Petrov This is my paper · ORCID

classification cs.CLcs.AI

keywords naturallanguageinferencecontrastsetsout-of-distributionrobustnessfine-tuningspuriouscorrelationsELECTRASNLIcorpuslinguistically-informedtransformations

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard benchmarks for natural language inference can overstate a model's understanding: ELECTRA-small scores 89.8% on SNLI but only 74.9% on a contrast set of minimally edited examples where the correct label flips. This paper tests whether a small dose of such contrastive examples, used for fine-tuning, can repair that gap. It finds that fine-tuning on 20% of an automatically generated contrast set raises accuracy on the held-out 80% to 90.7%, a gain of 16.8 points, while SNLI accuracy drops by only half a point. The paper argues that the gain comes from the model learning semantic distinctions instead of relying on word-overlap heuristics, and it matters because it suggests robustness can be bought with diverse training data rather than larger models.

What carries the argument

The load-bearing mechanism is the contrast set generated by Linguistically-Informed Transformations (LIT), a procedure that rewrites hypotheses in ways that preserve surface similarity but change the entailment label, exposing the model to exactly the kind of variation standard test sets miss. The paper splits the 14,363 generated examples into 20% for fine-tuning and 80% for evaluation, so the measured improvement reflects generalization to unseen perturbed inputs. The error taxonomy borrowed from prior stress-testing work—word overlap, negation, length mismatch, ambiguity, unknown—is what lets the paper attribute the accuracy gain to reduced reliance on superficial lexical patterns.

What would settle it

Re-generate the contrast set with a different perturbation sampler or with a train/test split that separates perturbation types, then fine-tune and evaluate again; if the 16.8-point gain disappears or splits unevenly across types, the original improvement came from learning the artifact of the generator rather than from more robust semantic understanding.

Watch

Extended reading notes

Core claim

The central claim is that exposing a pre-trained NLI model to a small number of contrast examples—minimal, meaning-changing perturbations of training sentences—during fine-tuning makes the model substantially more robust on out-of-distribution inputs without sacrificing in-distribution accuracy. The evidence is the 16.8-point jump on the untouched 80% of the contrast set, from 74.9% to 90.7%, alongside an SNLI accuracy change from 89.8% to 89.3%. Error analysis supports the mechanism: word-overlap errors drop from 35% to 10% of the mistakes, and negation errors halve, while the remaining errors are mostly cases the annotators themselves would find ambiguous.

Load-bearing premise

The method assumes that the automatically generated contrast set is a fair and representative sample of the meaning-changing variation a model should handle, and that the held-out 80% measures genuine robustness rather than memorization of perturbation patterns shared with the 20% used for fine-tuning.

Editorial extensions

If this is right

Using just 20% of a contrast set for fine-tuning recovers most of the out-of-distribution performance loss, implying that targeted data diversity is a cheap alternative to scaling model size.
Because SNLI accuracy barely moves (89.8% to 89.3%), the robustness gain is not bought at the cost of in-distribution performance.
The drop in word-overlap and negation errors after fine-tuning suggests the model is shifting away from shallow lexical heuristics toward label-relevant semantic cues.
The fact that performance levels off around 1,500 contrast examples (about 10% of the set) means a small, carefully chosen sample may be enough to get most of the benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

We infer that the same recipe would transfer to other NLI architectures (e.g., larger ELECTRA variants or decoder-only models) because the mechanism—reducing word-overlap shortcuts—is not architecture-specific, but the paper only demonstrates it on ELECTRA-small.
A stratified split that separates perturbation families (antonymy, negation, synonymy) would test whether the model learns general semantic invariance or just the specific transformation families seen in fine-tuning; the paper's random 20/80 split cannot distinguish these.
We infer that combining contrast fine-tuning with continued training on original SNLI examples could reduce the small 0.5-point SNLI dip while preserving the OOD gain, though the paper does not test this combination.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: short, readable empirical note, but the headline inference—that fine-tuning on contrast examples produces semantic understanding and out-of-distribution robustness—does not follow from the experimental design. The 16.8-point accuracy gain is real, but the held-out test is sampled from the same automatically generated contrast-set distribution, so the improvement likely reflects learning the perturbation statistics, not generalizable comprehension.

What the paper does well: it documents a clean phenomenon—ELECTRA-small gets 89.8% on SNLI, drops to 74.9% on LIT contrast sets—and shows that fine-tuning on 20% of those contrast examples lifts accuracy on the remaining 80% to 90.7% with only a 0.5-point drop on SNLI. That is a useful proof of concept for contrast-set-aware training. The error taxonomy, borrowed from Naik et al., is a sensible diagnostic, and the paper is honest enough to include a Limitations section conceding that a skewed label distribution could lead to majority-label prediction.

The soft spots are substantial. First, the evaluation is in-distribution relative to the contrast-set generator. A random 80/20 split of the same LIT set means the model can learn template-to-label correlations during fine-tuning (e.g., negation flips entailment to contradiction) and apply them to the held-out slice. That is not an out-of-distribution test. A proper transfer test would use contrast sets produced by a different generator, or at minimum fine-tune on matched random SNLI examples as a control. Second, the error analysis in Table 3 is based on 20 examples per condition; the reported shift from word-overlap to ambiguity errors is not statistically grounded. Third, the paper reports a single run, with no hyperparameters, seeds, code, or data, so the 16.8-point number has no error bar. The absence of these details alone prevents independent verification.

The title and framing overreach. 'Semantic understanding' is not established by adapting to a particular perturbation distribution. The paper itself includes the correct caveat in Limitations, but the abstract and conclusion ignore it. That disconnect is the paper's biggest problem.

Who this is for: readers working on contrast sets and NLI robustness, mainly as a motivating preliminary result. It does not deserve a full peer-review cycle as is; it needs an OOD transfer condition, a control, and proper reporting. If the authors add those, the revised version could be a solid workshop contribution or a short paper at a robustness workshop. As submitted, I would desk reject with an invitation to resubmit.

Referee Report

3 major / 5 minor

Summary. The paper studies the robustness of the ELECTRA-small model on natural language inference (NLI) when evaluated on contrast sets. Using the automatically generated Linguistically-Informed Transformations (LIT) contrast set, the author first shows that the model achieves 89.8% accuracy on SNLI but only 74.9% on the contrast set. The proposed remedy is to fine-tune the model on a small subset (20%) of the contrast set and evaluate on the remaining 80%. The author reports that fine-tuning raises contrast-set accuracy to 90.7% while SNLI accuracy remains essentially unchanged (89.3%), and interprets this as evidence that the model moves from relying on superficial patterns to semantic understanding. The paper also includes a small manual error analysis (20 examples per condition) suggesting that word-overlap and negation errors decrease while ambiguity errors increase.

Significance. If the central claim were supported by the experimental design, the paper would provide a simple and cheap recipe for improving robustness to contrast sets: fine-tune on a small sample of automatically generated perturbations. The reported effect size is large (16.8 points) and the observation that SNLI accuracy does not degrade is practically relevant. However, the contribution is incremental relative to the existing literature on contrast sets and robustness, and the paper does not release code or data. The main value would be as a proof of concept that small numbers of challenging examples can improve held-out performance, provided the held-out set is truly out-of-distribution. As it stands, the experimental design does not establish that claim, because the evaluation set is drawn from the same LIT-generated distribution as the fine-tuning set.

major comments (3)

[Section 4, Table 4] The central claim of improved out-of-distribution robustness is not supported by the evaluation protocol. The model is fine-tuned on a random 20% of the LIT contrast set and evaluated on the remaining 80% of the same automatically generated set. Both portions are produced by the same LIT transformation templates and therefore share the same distribution of perturbations and any template-to-label correlations (e.g., negation often flipping entailment to contradiction). The observed 16.8-point gain may thus reflect learning the LIT generator's artifacts rather than a general increase in semantic understanding. To support the paper's conclusion, the evaluation should include a transfer test on an independently constructed contrast set or a different type of perturbation (for example, HANS or a manually curated contrast set). Without such a control, the accuracy improvement is not evidence of robustness beyond the specific contrast-set distribution.
[Section 4, Table 3] The error-category analysis is based on only 20 incorrectly predicted examples before and after fine-tuning. With n=20, the reported shifts are within sampling variability: for a proportion of 0.35, the approximate 95% confidence interval is ±21 percentage points, so the change from 35% word-overlap errors to 10% is not statistically reliable, and the apparent increase in ambiguity errors (10% to 30%) carries similar uncertainty. The paper should either analyze a much larger sample of errors or report confidence intervals and significance tests before claiming that fine-tuning shifts the error profile from superficial patterns to genuine ambiguity.
[Section 3, Method] The fine-tuning procedure is not described in sufficient detail to reproduce or evaluate the result. The paper omits the learning rate, batch size, number of epochs, optimization hyperparameters, and random seed, and it reports only a single run. Figure 2 shows a learning curve over the number of contrast examples, but no variance or error bars are provided, and it is unclear whether all points use the same held-out split. The absence of code and data further limits reproducibility. The quantitative results (74.9% to 90.7%) should at minimum be accompanied by the training configuration and multiple seeds to establish that the improvement is not an artifact of one lucky split or run.

minor comments (5)

[Section 2.3] The term 'out-of-distribution' is used to describe the contrast set, but the contrast set is automatically generated from SNLI training examples by LIT transformations. The paper should clarify in what sense the contrast set is out-of-distribution relative to the SNLI test set, since the inputs are perturbed versions of in-distribution examples.
[Section 2.3] Figure 1 is introduced in Section 1 but never explicitly referenced in the text. The author should refer to Figure 1 where the contrast set examples are discussed.
[Section 2.1] There is a typo: 'pre-train transform networks' should be 'pre-train transformer networks'.
[Section 3] The description 'during validation of the model on the SNLI test set' is imprecise; validation and test are conflated. The paper should distinguish the model selection/validation step from the final evaluation on the test set.
[References] Several references have formatting issues (e.g., 'V . Le' with an extra space before the period, and 'Carolyn Rose' instead of 'Carolyn Rosé'). These should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the fine-tuning evaluation uses a held-out split; the same-distribution concern is an external-validity caveat, not a construction-level circularity.

full rationale

The paper's central experiment is an empirical fine-tuning study: ELECTRA-small is fine-tuned on 20% of a LIT-generated contrast set and evaluated on the remaining 80% (Sections 2.3 and 4). This is a standard training/held-out split; the held-out labels are not used in training, so the reported 90.7% accuracy is not a fitted value. There are no equations that reduce to one another, no parameter fitted to the evaluation data, and no load-bearing self-citations: all cited works (Gardner et al. 2020, Li et al. 2020, Naik et al. 2018) are external and not authored by the paper's author. The paper does claim the contrast set is out-of-distribution relative to SNLI; after fine-tuning on a subset of that same contrast set, the held-out portion is no longer fully out-of-distribution with respect to the fine-tuning data, so the gain may reflect learning the LIT perturbation distribution rather than general semantic understanding. That is a threat to external validity and should be discussed as a correctness or experimental-design risk, but it is not circularity in the derivation-theoretic sense used here: the evaluation does not reduce to the training input by construction. The Limitations section's note about skewed label distributions is a caveat, not a circular step. Overall, no significant circularity is present.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper relies on external resources (ELECTRA-small, SNLI, LIT) and introduces no new entities. The only hand-chosen number is the 20/80 split, which is not a fitted parameter. The main burden is the assumption that the automatically generated contrast set is a valid benchmark for out-of-distribution robustness.

free parameters (1)

Fine-tuning split ratio = 20% of 14,363 contrast examples (~2,873 examples)
The 20/80 split is chosen by hand. The learning curve in Figure 2 suggests diminishing returns around 1,500 examples, so the exact split affects the reported 90.7% held-out accuracy. This is a design choice, not fitted to optimize a separate metric.

assumptions (3)

domain assumption Pretrained ELECTRA-small is a representative model for studying NLI robustness.
The paper generalizes from a single model without testing other architectures or sizes. Section 2.1.
domain assumption LIT-generated contrast sets have correct gold labels and represent meaningful linguistic perturbations.
No manual verification of the full 14,363 examples is reported; the error analysis covers only 20 examples. Section 2.3.
domain assumption The fine-tuning and test portions of the contrast set are independent and identically distributed.
The split is random but no analysis is provided to show that the distribution of perturbation types is balanced across the split. Section 4.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets." pith.science (2026). https://pith.science/paper/LYZJZUUH

@misc{pith2026250102683,
  author       = {Pith},
  title        = {Pith review of: From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LYZJZUUH}},
  note         = {Machine review of arXiv:2501.02683}
}

read the original abstract

Large-scale pre-trained language models have demonstrated high performance on standard datasets for natural language inference (NLI) tasks. Unfortunately, these evaluations can be misleading, as although the models can perform well on in-distribution data, they perform poorly on out-of-distribution test sets, such as contrast sets. Contrast sets consist of perturbed instances of data that have very minor, but meaningful, changes to the input that alter the gold label, revealing how models can learn superficial patterns in the training data rather than learning more sophisticated language nuances. As an example, the ELECTRA-small language model achieves nearly 90% accuracy on an SNLI dataset but drops to 75% when tested on an out-of-distribution contrast set. The research carried out in this study explores how the robustness of a language model can be improved by exposing it to small amounts of more complex contrast sets during training to help it better learn language patterns. With this approach, the model recovers performance and achieves nearly 90% accuracy on contrast sets, highlighting the importance of diverse and challenging training data.

Figures

Figures reproduced from arXiv: 2501.02683 by the authors.

**Figure 2.** Performance improvement of the model on contrast sets with varying training sample sizes. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 5 linked inside Pith

[1]

Bowman, Gabor Angeli, Christopher Potts, and Christopher D

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. http://arxiv.org/abs/1508.05326 A large annotated corpus for learning natural language inference

arXiv 2015
[2]

Le, and Christopher D

Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. http://arxiv.org/abs/2003.10555 Electra: Pre-training text encoders as discriminators rather than generators

arXiv 2020
[3]

Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A

Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Walla...

arXiv 2020
[4]

Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkeld

Chuanrong Li, Lin Shengshuo, Leo Z. Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkeld. 2020. http://arxiv.org/abs/2010.08580 Linguistically-informed transformations (lit): A method for automatically generating contrast sets

arXiv 2020
[5]

Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. http://arxiv.org/abs/1806.00692 Stress test evaluation for natural language inference

arXiv 2018
[6]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[7]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Bowman, Gabor Angeli, Christopher Potts, and Christopher D

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. http://arxiv.org/abs/1508.05326 A large annotated corpus for learning natural language inference

arXiv 2015

[2] [2]

Le, and Christopher D

Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. http://arxiv.org/abs/2003.10555 Electra: Pre-training text encoders as discriminators rather than generators

arXiv 2020

[3] [3]

Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A

Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Walla...

arXiv 2020

[4] [4]

Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkeld

Chuanrong Li, Lin Shengshuo, Leo Z. Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkeld. 2020. http://arxiv.org/abs/2010.08580 Linguistically-informed transformations (lit): A method for automatically generating contrast sets

arXiv 2020

[5] [5]

Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. http://arxiv.org/abs/1806.00692 Stress test evaluation for natural language inference

arXiv 2018

[6] [6]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[7] [7]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...