{"id":"d474a90b-ffa1-4496-a4a1-20b0ceeac861","arxiv_id":"2501.02683","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuning ELECTRA-small on a 20% subsample of a contrast set raises held-out contrast set accuracy from 74.9% to 90.7% without hurting SNLI accuracy.","lead":"This paper fine-tunes a small language model on a subset of contrast-set examples and reports that accuracy on a held-out portion of the same contrast set jumps from 75% to 91%, while SNLI accuracy stays near 89%. It concludes that exposing models to diverse, challenging examples during training is key to robustness.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning and evaluation draw from the same LIT contrast-set distribution, so the 16.8-point gain may reflect learning contrast-set artifacts rather than semantic understanding; an out-of-distribution transfer test is needed.","rationale":"The reader's weakest assumption identifies essentially the same issue: the LIT contrast set may contain artifacts, and the held-out 80% of that set is not an independent out-of-distribution test. My stress-test sharpens this into a concrete mechanism: fine-tuning and testing on random splits of the same automatically generated set allows the model to learn perturbation-template-to-label correlations, which would inflate the headline gain without demonstrating semantic understanding. This does not make the reported accuracy numbers false; it makes the paper's central interpretive claim unsupported without additional validation. The paper is transparent about being a proof of concept and states its own limitation about skewed distributions, which is creditworthy. The appropriate verdict remains conditional: the direct empirical result is plausible, but the generalization claim needs an out-of-distribution transfer test before it can be accepted. No change to the reader's verdict is required.","tokens_in":4104,"tokens_out":3742,"duration_ms":39755,"concrete_test":"Generate a second LIT contrast set from a disjoint set of SNLI examples using transformation types that are held out from the 20% fine-tuning split, or alternatively evaluate on an independently constructed contrast set such as HANS. Fine-tune exactly as in Section 4 and measure accuracy on this unseen-transformation set. If the gain over the no-fine-tuning baseline largely disappears, the reported 90.7% is distribution adaptation to the contrast-set generator; if the gain transfers, the semantic-understanding interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 reports accuracy on the contrast set rising from 74.9% to 90.7% after fine-tuning on 20% of the LIT-generated contrast set and evaluating on the remaining 80%. The central claim interprets this as improved semantic understanding and robustness. The load-bearing assumption is that the held-out 80% is a valid out-of-distribution test. It is not: it is a random split from the same automatically generated contrast set, produced by the same LIT transformation templates and likely sharing the same annotation artifacts. If LIT transformations systematically correlate with label changes (for example, adding negation to a hypothesis often flips entailment to contradiction), the model can learn template-to-label rules during fine-tuning and apply them to held-out examples. The observed improvement would then reflect adaptation to the contrast-set generator's distribution, not generalizable semantic competence. The paper's own Limitations section admits that a skewed example or label distribution could cause the model to predict the majority label, and automatic generation makes such skew plausible. In addition, Table 3's error-category shift is based on only 20 examples per condition, so the claimed change from predominantly 'word overlap' errors to 'ambiguity' errors is not statistically grounded. The accuracy gain itself is not in dispute, but the inference from that gain to semantic understanding is not supported by the current experimental design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the robustness of the ELECTRA-small model on natural language inference (NLI) when evaluated on contrast sets. Using the automatically generated Linguistically-Informed Transformations (LIT) contrast set, the author first shows that the model achieves 89.8% accuracy on SNLI but only 74.9% on the contrast set. The proposed remedy is to fine-tune the model on a small subset (20%) of the contrast set and evaluate on the remaining 80%. The author reports that fine-tuning raises contrast-set accuracy to 90.7% while SNLI accuracy remains essentially unchanged (89.3%), and interprets this as evidence that the model moves from relying on superficial patterns to semantic understanding. The paper also includes a small manual error analysis (20 examples per condition) suggesting that word-overlap and negation errors decrease while ambiguity errors increase.","tokens_in":4449,"tokens_out":3209,"duration_ms":31763,"significance":"If the central claim were supported by the experimental design, the paper would provide a simple and cheap recipe for improving robustness to contrast sets: fine-tune on a small sample of automatically generated perturbations. The reported effect size is large (16.8 points) and the observation that SNLI accuracy does not degrade is practically relevant. However, the contribution is incremental relative to the existing literature on contrast sets and robustness, and the paper does not release code or data. The main value would be as a proof of concept that small numbers of challenging examples can improve held-out performance, provided the held-out set is truly out-of-distribution. As it stands, the experimental design does not establish that claim, because the evaluation set is drawn from the same LIT-generated distribution as the fine-tuning set.","major_comments":[{"comment":"The central claim of improved out-of-distribution robustness is not supported by the evaluation protocol. The model is fine-tuned on a random 20% of the LIT contrast set and evaluated on the remaining 80% of the same automatically generated set. Both portions are produced by the same LIT transformation templates and therefore share the same distribution of perturbations and any template-to-label correlations (e.g., negation often flipping entailment to contradiction). The observed 16.8-point gain may thus reflect learning the LIT generator's artifacts rather than a general increase in semantic understanding. To support the paper's conclusion, the evaluation should include a transfer test on an independently constructed contrast set or a different type of perturbation (for example, HANS or a manually curated contrast set). Without such a control, the accuracy improvement is not evidence of robustness beyond the specific contrast-set distribution.","section":"Section 4, Table 4"},{"comment":"The error-category analysis is based on only 20 incorrectly predicted examples before and after fine-tuning. With n=20, the reported shifts are within sampling variability: for a proportion of 0.35, the approximate 95% confidence interval is ±21 percentage points, so the change from 35% word-overlap errors to 10% is not statistically reliable, and the apparent increase in ambiguity errors (10% to 30%) carries similar uncertainty. The paper should either analyze a much larger sample of errors or report confidence intervals and significance tests before claiming that fine-tuning shifts the error profile from superficial patterns to genuine ambiguity.","section":"Section 4, Table 3"},{"comment":"The fine-tuning procedure is not described in sufficient detail to reproduce or evaluate the result. The paper omits the learning rate, batch size, number of epochs, optimization hyperparameters, and random seed, and it reports only a single run. Figure 2 shows a learning curve over the number of contrast examples, but no variance or error bars are provided, and it is unclear whether all points use the same held-out split. The absence of code and data further limits reproducibility. The quantitative results (74.9% to 90.7%) should at minimum be accompanied by the training configuration and multiple seeds to establish that the improvement is not an artifact of one lucky split or run.","section":"Section 3, Method"}],"minor_comments":[{"comment":"The term 'out-of-distribution' is used to describe the contrast set, but the contrast set is automatically generated from SNLI training examples by LIT transformations. The paper should clarify in what sense the contrast set is out-of-distribution relative to the SNLI test set, since the inputs are perturbed versions of in-distribution examples.","section":"Section 2.3"},{"comment":"Figure 1 is introduced in Section 1 but never explicitly referenced in the text. The author should refer to Figure 1 where the contrast set examples are discussed.","section":"Section 2.3"},{"comment":"There is a typo: 'pre-train transform networks' should be 'pre-train transformer networks'.","section":"Section 2.1"},{"comment":"The description 'during validation of the model on the SNLI test set' is imprecise; validation and test are conflated. The paper should distinguish the model selection/validation step from the final evaluation on the test set.","section":"Section 3"},{"comment":"Several references have formatting issues (e.g., 'V . Le' with an extra space before the period, and 'Carolyn Rose' instead of 'Carolyn Rosé'). These should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short empirical study with a plausible but currently under-supported central claim. The main issue is the lack of a genuine out-of-distribution evaluation: fine-tuning and testing on random splits of the same automatically generated contrast set makes the 16.8-point gain difficult to interpret. The error analysis is too small to bear the weight of the qualitative conclusions. The missing training details and lack of code/data also hurt reproducibility. I would encourage the editor to request a major revision that adds a transfer experiment and a more rigorous error analysis, rather than rejecting outright, because the underlying question is worth addressing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: short, readable empirical note, but the headline inference—that fine-tuning on contrast examples produces semantic understanding and out-of-distribution robustness—does not follow from the experimental design. The 16.8-point accuracy gain is real, but the held-out test is sampled from the same automatically generated contrast-set distribution, so the improvement likely reflects learning the perturbation statistics, not generalizable comprehension.\n\nWhat the paper does well: it documents a clean phenomenon—ELECTRA-small gets 89.8% on SNLI, drops to 74.9% on LIT contrast sets—and shows that fine-tuning on 20% of those contrast examples lifts accuracy on the remaining 80% to 90.7% with only a 0.5-point drop on SNLI. That is a useful proof of concept for contrast-set-aware training. The error taxonomy, borrowed from Naik et al., is a sensible diagnostic, and the paper is honest enough to include a Limitations section conceding that a skewed label distribution could lead to majority-label prediction.\n\nThe soft spots are substantial. First, the evaluation is in-distribution relative to the contrast-set generator. A random 80/20 split of the same LIT set means the model can learn template-to-label correlations during fine-tuning (e.g., negation flips entailment to contradiction) and apply them to the held-out slice. That is not an out-of-distribution test. A proper transfer test would use contrast sets produced by a different generator, or at minimum fine-tune on matched random SNLI examples as a control. Second, the error analysis in Table 3 is based on 20 examples per condition; the reported shift from word-overlap to ambiguity errors is not statistically grounded. Third, the paper reports a single run, with no hyperparameters, seeds, code, or data, so the 16.8-point number has no error bar. The absence of these details alone prevents independent verification.\n\nThe title and framing overreach. 'Semantic understanding' is not established by adapting to a particular perturbation distribution. The paper itself includes the correct caveat in Limitations, but the abstract and conclusion ignore it. That disconnect is the paper's biggest problem.\n\nWho this is for: readers working on contrast sets and NLI robustness, mainly as a motivating preliminary result. It does not deserve a full peer-review cycle as is; it needs an OOD transfer condition, a control, and proper reporting. If the authors add those, the revised version could be a solid workshop contribution or a short paper at a robustness workshop. As submitted, I would desk reject with an invitation to resubmit.","headline":"Contrast-set fine-tuning recovers accuracy on the same contrast-set distribution, but the 'semantic understanding' framing doesn't survive contact with the evaluation design.","tokens_in":4879,"tokens_out":3430,"would_cite":false,"duration_ms":32556,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a pre-trained NLI model on a small subset of contrast examples lifts its accuracy on held-out contrast data from 74.9% to 90.7% while leaving standard test accuracy almost unchanged.","keywords":["natural language inference","contrast sets","out-of-distribution robustness","fine-tuning","spurious correlations","ELECTRA","SNLI corpus","linguistically-informed transformations"],"falsifier":"Re-generate the contrast set with a different perturbation sampler or with a train/test split that separates perturbation types, then fine-tune and evaluate again; if the 16.8-point gain disappears or splits unevenly across types, the original improvement came from learning the artifact of the generator rather than from more robust semantic understanding.","tokens_in":3861,"feed_emoji":"🧠","tokens_out":6014,"duration_ms":50245,"temperature":0.7,"pith_summary":"Standard benchmarks for natural language inference can overstate a model's understanding: ELECTRA-small scores 89.8% on SNLI but only 74.9% on a contrast set of minimally edited examples where the correct label flips. This paper tests whether a small dose of such contrastive examples, used for fine-tuning, can repair that gap. It finds that fine-tuning on 20% of an automatically generated contrast set raises accuracy on the held-out 80% to 90.7%, a gain of 16.8 points, while SNLI accuracy drops by only half a point. The paper argues that the gain comes from the model learning semantic distinctions instead of relying on word-overlap heuristics, and it matters because it suggests robustness can be bought with diverse training data rather than larger models.","feed_headline":"Small contrast set fine-tuning lifts robustness score 16.8 points","feed_subtitle":"A model that scored 74.9% on held-out contrast examples reaches 90.7% after training on just 20% of them.","key_machinery":"The load-bearing mechanism is the contrast set generated by Linguistically-Informed Transformations (LIT), a procedure that rewrites hypotheses in ways that preserve surface similarity but change the entailment label, exposing the model to exactly the kind of variation standard test sets miss. The paper splits the 14,363 generated examples into 20% for fine-tuning and 80% for evaluation, so the measured improvement reflects generalization to unseen perturbed inputs. The error taxonomy borrowed from prior stress-testing work—word overlap, negation, length mismatch, ambiguity, unknown—is what lets the paper attribute the accuracy gain to reduced reliance on superficial lexical patterns.","core_discovery":"The central claim is that exposing a pre-trained NLI model to a small number of contrast examples—minimal, meaning-changing perturbations of training sentences—during fine-tuning makes the model substantially more robust on out-of-distribution inputs without sacrificing in-distribution accuracy. The evidence is the 16.8-point jump on the untouched 80% of the contrast set, from 74.9% to 90.7%, alongside an SNLI accuracy change from 89.8% to 89.3%. Error analysis supports the mechanism: word-overlap errors drop from 35% to 10% of the mistakes, and negation errors halve, while the remaining errors are mostly cases the annotators themselves would find ambiguous.","pith_inferences":["We infer that the same recipe would transfer to other NLI architectures (e.g., larger ELECTRA variants or decoder-only models) because the mechanism—reducing word-overlap shortcuts—is not architecture-specific, but the paper only demonstrates it on ELECTRA-small.","A stratified split that separates perturbation families (antonymy, negation, synonymy) would test whether the model learns general semantic invariance or just the specific transformation families seen in fine-tuning; the paper's random 20/80 split cannot distinguish these.","We infer that combining contrast fine-tuning with continued training on original SNLI examples could reduce the small 0.5-point SNLI dip while preserving the OOD gain, though the paper does not test this combination."],"forward_implications":["Using just 20% of a contrast set for fine-tuning recovers most of the out-of-distribution performance loss, implying that targeted data diversity is a cheap alternative to scaling model size.","Because SNLI accuracy barely moves (89.8% to 89.3%), the robustness gain is not bought at the cost of in-distribution performance.","The drop in word-overlap and negation errors after fine-tuning suggests the model is shifting away from shallow lexical heuristics toward label-relevant semantic cues.","The fact that performance levels off around 1,500 contrast examples (about 10% of the set) means a small, carefully chosen sample may be enough to get most of the benefit."],"supporting_citations":[{"why":"Supplies the ELECTRA-small model architecture and pretrained weights used throughout the experiments.","marker":"Clark et al., 2020"},{"why":"Provides the SNLI corpus, the in-distribution training and test data for NLI.","marker":"Bowman et al., 2015"},{"why":"Introduces contrast sets as a method for evaluating local decision boundaries, motivating the robustness gap the paper targets.","marker":"Gardner et al., 2020"},{"why":"Provides the Linguistically-Informed Transformations (LIT) procedure used to automatically generate the contrast set.","marker":"Li et al., 2020"},{"why":"Supplies the error taxonomy (word overlap, negation, length mismatch, ambiguity, unknown) used in the error analysis.","marker":"Naik et al., 2018"}],"fun_headline_variants":["Contrast-set fine-tuning boosts OOD accuracy 16.8 points","20% contrast data lifts OOD robustness to 90.7%","16.8-point OOD robustness jump from contrast-set fine-tuning","Fine-tuning on small contrast set: 74.9% to 90.7% OOD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the automatically generated contrast set is a fair and representative sample of the meaning-changing variation a model should handle, and that the held-out 80% measures genuine robustness rather than memorization of perturbation patterns shared with the 20% used for fine-tuning.","fun_headline_variants_meta":{"raw":{"variants":["Contrast-set fine-tuning boosts OOD accuracy 16.8 points","20% contrast data lifts OOD robustness to 90.7%","16.8-point OOD robustness jump from contrast-set fine-tuning","Fine-tuning on small contrast set: 74.9% to 90.7% OOD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001266,"raw_usage":{"total_tokens":5150,"prompt_tokens":884,"completion_tokens":4266,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":4164}},"tokens_in":500,"tokens_out":4266,"duration_ms":34743,"temperature":1.0,"reasoning_tokens":4164,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:06:51.990289+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-generate the contrast set with a different perturbation sampler or with a train/test split that separates perturbation types, then fine-tune and evaluate again; if the 16.8-point gain disappears or splits unevenly across types, the original improvement came from learning the artifact of the generator rather than from more robust semantic understanding.","supporting_citations":[],"review_version":1}