{"id":"f192d654-06b0-46e9-9c48-e92f8f76077b","arxiv_id":"2510.24295","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A new automatic method for creating minimal reasoning-preserving variants of NLI problems shows that 14 models drop 4 to 20 percent in accuracy on those variants.","lead":"This paper tests whether AI language models still reason correctly when a few words in a reasoning problem are swapped for similar words. Most models answered fewer of the lightly edited problems correctly, showing they rely on surface wording more than on the underlying logic.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Q: Variant label preservation is only spot-checked; if the true label-inheritance failure rate is 5–10%, the reported 4–9% pattern-accuracy drop may be an artifact.","rationale":"The reader's weakest_assumption is exactly the step I would attack first: label inheritance. Variant accuracy is only meaningful if the copied label is right; otherwise models are being scored against labels that are not entailed. The paper has partial safeguards—manual annotation, BART exclusion, the 91% good post-exclusion score—so I do not call the claim unsupported. But the annotation sample is small relative to the 5,781 variants in Table 2, it mixes fluency with reasoning in a single score, and no agreement statistics are given. The paper's own footnote 4 and §5.5 show label correctness is not guaranteed, so this is a real empirical risk, not a hypothetical one. I agree with the reader's CONDITIONAL verdict and would not move it. A secondary issue is the abstract's promise of 'two widely used existing ones' while the experiments describe only SNLI; that is a reporting/scoping problem but less central than label integrity.","tokens_in":20302,"tokens_out":6368,"duration_ms":57476,"concrete_test":"Release the final variant set with provenance and have two independent annotators score label preservation (the R component of the paper's own scale) on a stratified random sample of at least 1,000 variants, oversampling seeds with borderline pattern accuracy (0.85–0.95). Then recompute QT (PA at 90%) and MT using only variants with confirmed label preservation (or with corrected labels). If the 4–9% drop and MT≈60% survive, the concern is settled; if the drop shrinks below about 2 points or MT rises above 75%, the robustness claim needs to be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every variant ⟨P_ij,H_ij,l⟩ inherits the seed's gold label in §3, so all reported sample/pattern accuracies assume the replacement preserves the entailment relation. The only checks are 300 annotated variants (100 per POS class) plus 100 post-BART, with no inter-annotator agreement reported; footnote 4 explicitly concedes that variants with incorrect labels are still possible. At QT=90%, a seed with 20 variants tolerates only two errors, so even a 5–10% label-preservation failure rate would independently make a non-trivial fraction of seeds fail pattern accuracy, inflating the headline 4–9% drop and the MT≈60% claim. The paper's own §5.5 shows seed labels are unreliable (only 29% of the 31 hardest seeds have a correct gold label), so the assumption cannot be secured by appealing to seed quality. BART exclusion reduces but does not remove the threat, and the final variant dataset is not released.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MERGE, a methodology for constructing minimally altered NLI problems by replacing open-class words shared between premise and hypothesis with higher-probability, same-POS suggestions from a panel of MLMs. Variants inherit the seed label, and the filtering is designed to preserve word overlap and syntax. The authors evaluate 14 NLI models on variants derived from SNLI test problems, using both sample accuracy and pattern accuracy (a per-seed threshold-consistency metric). They report that at a 90% quality threshold, pattern accuracy drops by roughly 4-9 percentage points relative to seed accuracy, and that matching seed-level pattern accuracy requires lowering the threshold to about 60% for most models. Additional analyses examine the effect of replacement POS class, the origin MLM, and different filtering criteria.","tokens_in":20668,"tokens_out":9074,"duration_ms":81583,"significance":"If the label-preservation assumption is adequately validated, MERGE is a practical and low-cost generalization test: it preserves word overlap and syntax by construction, and it explicitly checks a common confound in automatic contrast-set generation, namely whether NLI models are favored when variants come from their own MLM family (Section 5.3). The empirical result that strong NLI models are not robust to minimal, label-preserving lexical substitutions would be a useful data point for the robustness literature. The paper is also candid about limitations, including the possibility of label-changing variants (footnote 4) and unreliable seed labels in SNLI (Section 5.5). A notable weakness is that the final variant dataset is not released, which limits independent verification of the central label-preservation claim.","major_comments":[{"comment":"The entire evaluation treats every variant <P_ij, H_ij, l> as inheriting the seed label l. The only direct validation is a spot-check of 300 variants (100 per POS class) plus 100 post-BART variants, annotated by two authors with no inter-annotator agreement reported, and footnote 4 concedes that label-changing variants are still possible. Because pattern accuracy at QT=90% with 20 variants per seed allows at most two errors before a seed fails, a label-preservation failure rate of even a few percent can push a non-trivial fraction of seeds over the threshold and directly inflate the reported QT and MT values. Section 5.5 shows that seed labels themselves are noisy (only 29% of the 31 hardest seeds have a correct gold label), so the assumption cannot be secured by appealing to seed quality. I therefore ask for a larger random human-validation sample with agreement statistics, or a release of the final variants so that label preservation can be independently audited.","section":"Section 3, Eq. (1); Section 4, Variants Manual Annotation"},{"comment":"The headline 4-9% drop is computed as SA on ALLSeed minus PA on ALLVar at the 90% threshold, which mixes a per-example metric with a per-seed threshold metric. The per-example accuracy difference between ALLSeed and ALLVar is much smaller in Table 3 (e.g., 89.6 vs. 88.9 for BERT-B-S and 90.5 vs. 89.1 for OPT-1.3B). I do not dispute that pattern accuracy is a useful consistency metric, but the paper should state explicitly that the 4-9% figure is a pattern-accuracy drop rather than a sample-accuracy drop, and the abstract's wording 'perform 4-20% worse on variants' should be revised to name the metric.","section":"Section 5.1, Table 3"}],"minor_comments":[{"comment":"The table is hard to parse because adjacent values are concatenated without separators (e.g., '88.9-4.959'); please format the columns properly.","section":"Table 3"},{"comment":"The abstract's '4-20%' range should specify that the lower bound is the pattern-accuracy drop at QT and the upper bound occurs at stricter thresholds; the body reports 4-9% at QT.","section":"Abstract"},{"comment":"The comparison between filtering criteria is confounded by dataset size: P_union_H, POS, and None have about 380k unique variants each while ALLVar has about 190k. The conclusion that 'the number of variants seems more important' should be supported by a size-matched subsample.","section":"Section 5.4"},{"comment":"The definition of Multi-MLM as 'any two MLMs, potentially including the evaluated model' is ambiguous; clarify whether the evaluated model's own MLM can be one of the two, since this affects the interpretation of the Multi-MLM curve.","section":"Section 5.3"},{"comment":"The re-annotation of the 31 hard seeds is done by the authors and used to argue that seed labels are unreliable; reporting annotation agreement or a second annotator would strengthen this point.","section":"Section 5.5"},{"comment":"Please provide a reproducibility statement with code and dataset release information; the current version does not state whether the variant dataset will be made available.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The main risk is not circularity: Section 5.3 directly addresses the MLM-backbone confound, and the paper is careful not to claim a fitted derivation. The load-bearing issue is the spot-checked label-inheritance assumption, compounded by the unreleased dataset. If the authors can supply a larger validation sample or release the variants, I would be inclined to support acceptance after the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The MERGE paper is a genuine methodological contribution, not just another contrast set. The specific pipeline — replacing shared open-class words in both premise and hypothesis with same-POS, higher-probability MLM suggestions, inheriting the seed label, and scoring by pattern accuracy — is new as a combination, and it is a sensible way to isolate lexical-surface sensitivity from syntax and word-overlap confounds. The paper also does the right diligence: it tests whether the generating MLMs favor matching NLI models, checks filtering criteria separately, reports paired significance tests, and even digs into the hardest seeds and finds that many SNLI gold labels are wrong. That last analysis is honest and useful. The 4–9% drop at the 90% threshold and the roughly 60% matching-threshold claim are internally consistent with the tables and figures, and the qualitative story — models are brittle to minimal, reasoning-preserving changes — fits the broader literature.\n\nNow the soft spots, in proportion. The biggest one is the one the stress-test flags: label inheritance. Every variant is assumed to preserve the seed label, but only a few hundred variants are manually checked, with no inter-annotator agreement reported, and footnote 4 concedes errors are possible. The stress-test arithmetic is fair: at QT=90% with 20 variants per seed, a 5–10% label-preservation failure rate could independently push seeds below threshold and inflate the headline drop. I want to be clear that this does not sink the paper — the manual checks and BART exclusion suggest the failure rate is probably low — but it does mean the precise magnitude of the drop is not yet fully secured. The fix is straightforward: release the variant set and have a small number of annotators validate a larger sample, or at least report the current annotation agreement and bound the possible inflation.\n\nOther issues are minor. The abstract says \"two widely used existing datasets\" but the experiments only use SNLI; that mismatch should be fixed. Some of the filtering analyses (e.g., Section 5.4) are descriptive and would benefit from significance testing. The dataset and code are not released, which matters for reproducibility. None of these are load-bearing.\n\nWho is this for? Anyone building or evaluating NLI benchmarks, and anyone who cares about whether contrast-set drops reflect reasoning failures or artifact. A serious referee should engage with it. The empirical core is plausible, the methodology is reusable, and the limitations are mostly acknowledged in the text. I would recommend peer review with the label-inheritance validation as the main requested revision.","headline":"A useful, mostly sound NLI robustness study whose headline gap is plausible but whose label-inheritance assumption needs fuller validation or qualification before the numbers are load-bearing.","tokens_in":21080,"tokens_out":651,"would_cite":true,"duration_ms":7768,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NLI models lose 4–9 percent on minimal word swaps","keywords":["natural language inference","generalization testing","minimal edits","contrast sets","masked language models","robustness","pattern accuracy"],"falsifier":"Re-annotate a random sample of, say, 500 MERGE variants with fresh annotators who see no seed label, and compare their labels to the inherited seed labels; if more than about 10 percent of variants receive a different label, the reported 4–9 percent accuracy drop is substantially contaminated by label-inheritance errors. A complementary check is to recompute the pattern-accuracy curves after dropping variants whose seed problems the paper itself identifies as unreliable.","tokens_in":20137,"feed_emoji":"🔁","tokens_out":7843,"duration_ms":58793,"temperature":0.7,"pith_summary":"This paper claims that state-of-the-art natural language inference (NLI) models are not robust to even the mildest, reasoning-preserving surface changes. The authors propose MERGE, a test that creates many variants of each original NLI problem by replacing open-class words shared by the premise and hypothesis with more probable, same-part-of-speech substitutes suggested by masked language models. Because the variants keep the original reasoning, word overlap, and syntax, any drop in accuracy is attributed to a real generalization failure rather than to a shift in test conditions. Across fourteen models, they report that at a 90 percent variant-quality threshold, sample accuracy falls by about 4–9 percent relative to seed problems, and reaching seed-level pattern accuracy requires lowering the accuracy threshold to roughly 60 percent for most models.","feed_headline":"NLI models lose 4–9 percent on minimal word swaps","feed_subtitle":"Reasoning-preserving variants trip 14 models; seed-level scores need a ~60 percent threshold.","key_machinery":"The central mechanism is the MERE generation pipeline: for each seed problem, every open-class word shared between premise and hypothesis is masked in both sentences, and a variant word is kept only if it is suggested by at least one masked language model at every occurrence of that word in both premise and hypothesis, is more probable than the original word under that model, shares the same part of speech, and does not occur elsewhere in the problem. This intersection rule is what guarantees that the replacement is equally felicitous in premise and hypothesis, thereby preserving the seed's reasoning and word overlap. Pattern accuracy over at least 20 variants per seed is then the evaluation device that turns individual predictions into a consistency measure, and thresholds such as 90 percent define what counts as robust.","core_discovery":"The central discovery is that NLI models generalize poorly under the simplest possible generalization test: replacing a shared open-class word in the premise and hypothesis with a more probable, same-part-of-speech word that preserves the original logical relation. The paper builds MERGE by masking each shared word in both premise and hypothesis, collecting suggestions from several masked language models, and keeping only suggestions that are more probable than the original word, belong to the same part of speech, and are validated at every occurrence of the word in both sentences. The resulting variants are scored with pattern accuracy, which credits a seed problem only when a model correctly classifies at least an x fraction of its variants. At the chosen 90 percent threshold, models are 4–9 percent less accurate on variants than on the original seed problems, and most models need a 60 percent accuracy threshold to match their seed-level pattern accuracy. The authors also find that noun replacements are harder than verb replacements, that the MLM used to generate variants does not systematically favor its NLI counterpart, and that the sheer number of unique variants affects scores more than stricter filtering for plausibility.","pith_inferences":["If MERGE's reasoning-preservation claim is right, the 4–9 percent drop is a lower bound on fragility: adversarial or distribution-shifting tests would likely show larger gaps, so robustness research should use MERGE as a baseline rather than a worst case.","Because the variant space is defined by what the selected masked language models find plausible, the absolute accuracy numbers are partly a property of the generation pipeline; a different MLM set could change the difficulty ranking of the same models.","A direct test of the paper's token-frequency observation would correlate per-variant accuracy with the frequency of the replacement word in the model's training corpus; the paper's Section 5.5 hints that rare replacements drive part of the drop for SNLI-only models.","Fine-tuning on MERGE variants (data augmentation) is an untested extension; with roughly 190k unique variants the experiment is feasible and would show whether the observed fragility can be trained away."],"forward_implications":["Standard in-distribution NLI accuracy systematically overstates reasoning ability: models scoring above 90 percent on SNLI fall 4–9 percent on these minimally altered, reasoning-preserving variants.","Pattern accuracy at a threshold is a more informative evaluation than sample-level accuracy, because it reveals when a model fails whole problem families even while getting most individual variants right.","NLI models lean on surface form, since swapping words that are irrelevant to the inference, such as changing 'small' to 'little' in 'A small girl carries a girl', produces measurable instability.","Robustness-test design should control the number of unique variants, because lexical diversity affects model scores more than stricter plausibility filters.","The MERGE methodology is model- and dataset-agnostic, so it can be applied to other NLI benchmarks and to other natural-language tasks that can be framed as NLI."],"supporting_citations":[{"why":"Supplies the Pattern Accuracy metric that MERGE uses to score consistency across variants of a seed problem.","marker":"Abzianidze et al., 2023"},{"why":"Pioneered automatic word-replacement contrast sets for NLI; MERGE extends this by preserving the seed label and word overlap.","marker":"Glockner et al., 2018"},{"why":"Prior contrast-set study showing NLI models' fragility; MERGE is designed to avoid its confounds of changed syntax and overlap.","marker":"Li et al., 2020"},{"why":"Provides the contrast-set evaluation philosophy that motivates testing models on minimal local changes.","marker":"Gardner et al., 2020"},{"why":"Recent paraphrastic-robustness baseline that MERGE compares against on preservation of reasoning and word overlap.","marker":"Verma et al., 2023"},{"why":"Source of the SNLI seed problems from which MERGE variants are generated.","marker":"Bowman et al., 2015"}],"fun_headline_variants":["Minimal word swaps expose NLI generalization gap","NLI models stumble on simple synonym swaps","Reasoning models fail minimal expression test","Swap one word, NLI accuracy drops 4–9%","NLI robustness test: 14 models tripped by tiny edits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every variant produced by replacing a shared word with a more probable, same-part-of-speech word inherits the seed problem's entailment label; the paper manually spot-checks only a few hundred examples and notes in a footnote that incorrect label inheritance is possible.","fun_headline_variants_meta":{"raw":{"variants":["Minimal word swaps expose NLI generalization gap","NLI models stumble on simple synonym swaps","Reasoning models fail minimal expression test","Swap one word, NLI accuracy drops 4–9%","NLI robustness test: 14 models tripped by tiny edits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000603,"raw_usage":{"total_tokens":2834,"prompt_tokens":987,"completion_tokens":1847,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1771}},"tokens_in":603,"tokens_out":1847,"duration_ms":11688,"temperature":1.0,"reasoning_tokens":1771,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:42:14.963904+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of, say, 500 MERGE variants with fresh annotators who see no seed label, and compare their labels to the inherited seed labels; if more than about 10 percent of variants receive a different label, the reported 4–9 percent accuracy drop is substantially contaminated by label-inheritance errors. A complementary check is to recompute the pattern-accuracy curves after dropping variants whose seed problems the paper itself identifies as unreliable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Pattern Accuracy metric that MERGE uses to score consistency across variants of a seed problem."}],"review_version":2}