{"id":"ebdb735f-a757-4890-8823-618ac81630e1","arxiv_id":"2506.15239","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Language models are significantly worse at natural language inference when sentences are written in Basque or Spanish regional dialects, especially for Basque.","lead":"This paper introduces XNLIvar, a new hand-made dataset that rewrites standard Basque and Spanish sentences into regional dialects, and tests AI models on understanding whether short statements logically follow from each other. It finds that language models make more mistakes on dialectal versions, especially for Basque, which matters because AI should serve all speakers, not just those using the standard form.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The p<.001 chi-square result reported in §6 does not reproduce from the counts in Table 6; the statistical significance underpinning the headline claim is internally inconsistent.","rationale":"The reader's verdict is CONDITIONAL and I do not propose rejecting the paper, but the strongest claim as stated contains a reproducibility defect that is more objective than the annotator-authenticity concern. The paper's own Table 6 provides enough information to see that a chi-square test of independence on the 'No repetitions' rows cannot yield p<.001; the discrepancy is large (Basque p≈0.12, Spanish p≈0.5). Unless a paired test was used and mislabeled as a chi-square test of independence, the paper's only significance evidence is wrong. This matters because the central claim is not merely that accuracy drops (it visibly does) but that the drop is statistically significant and attributable to variation; the latter inference is only as strong as the test behind it. I would keep the verdict CONDITIONAL but require, as an explicit condition, that the authors either release the per-instance predictions and correct the significance analysis or supply the discordant-pair table for a McNemar test. The reader's weakest assumption about dialect authenticity remains relevant, but it is not the same as the internal statistical inconsistency identified here.","tokens_in":16653,"tokens_out":12731,"duration_ms":140927,"concrete_test":"Recompute the significance test from the released XNLIvar predictions or, if predictions are not released, from the exact per-instance predictions behind Table 6. Build the 2x2 contingency table for each 'No repetitions' split and compute both Pearson's chi-square and McNemar's test with the full discordant-pair count. If p<.001 cannot be reproduced by either test, the reported significance claim is false and the headline conclusion must be revised; if McNemar is the intended analysis, the authors must publish the discordance matrix so the p-value can be checked.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that dialectal variants cause a statistically significant drop in NLI accuracy depends entirely on the chi-square tests reported in Section 6 for the 'No repetitions' splits. The counts implied by Table 6 are incompatible with the reported p-values. For Basque, the No repetitions row shows 71.77% (≈446/621) for XNLIeuvar versus 75.63% (≈470/621) for XNLIeunative; a standard 2x2 chi-square test of independence on these margins gives χ²≈2.4, p≈0.12. For Spanish, the row shows 73.00% (≈453/621) versus 74.61% (≈463/621), giving χ²≈0.4, p≈0.5. Neither is p<.001. A paired McNemar test could in principle yield a smaller p-value, but the paper reports a 'chi-square test of independence' with df=1 and supplies no discordant-pair counts, so the test as described is either misapplied or misreported. Since Section 5.1's causal language ('results show that language models perform worse when variants are included') and Section 6's 'due to linguistic variation' conclusion rest on this significance claim, the headline result is not established by the paper's own data. Independently of this, the variant labels are never verified to preserve the gold NLI relation, which is a second confound; the p-value problem is the more immediately checkable defect.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents XNLIvar, a manually curated parallel NLI dataset that adds Basque and Spanish geographic variants to the XNLIeunative resource. It evaluates encoder-only models (XLM-RoBERTa, mDeBERTa) in model-transfer, translate-train, and translate-test settings, and decoder-only LLMs (Llama-3.1, Gemma 2) in several prompting formulations. The main empirical claim is that accuracy drops when dialectal variants are included, especially in Basque, and that this drop reflects linguistic variation rather than lexical overlap; Western Basque is singled out as particularly challenging. The authors also report chi-square tests in Section 6 claiming highly significant differences on parallel 'No repetitions' subsets.","tokens_in":16900,"tokens_out":9327,"duration_ms":79621,"significance":"If the empirical claims held up, this would be a valuable contribution: the dataset is, to my knowledge, the first manually curated variation-inclusive NLI benchmark for these language varieties, and the evaluation covers a sensible range of model families and training regimes. The public release of data and code is commendable. However, the headline statistical evidence is internally inconsistent: the reported p-values cannot be reproduced from the accuracies in Table 6. There are also unaddressed confounds related to label preservation and the selection of the 'No repetitions' subset. The resource itself is potentially useful, but the central claim about the cause and significance of the observed drop is not currently established by the data as presented.","major_comments":[{"comment":"The chi-square tests reported for the 'No repetitions' splits are not reproducible from the table. For Basque, accuracies of 75.63% (≈470/621) and 71.77% (≈446/621) yield a 2x2 chi-square statistic of about 2.39 (p≈0.12), not p<.001. For Spanish, accuracies of 74.61% (≈463/621) and 73.00% (≈453/621) yield χ²≈0.42 (p≈0.52). The generative Spanish comparison (77.29 vs 77.13 on 621 items) is trivially non-significant. The claim in Section 5.1 that 'language models perform worse when variants are included' and the Section 6 conclusion that the decline is 'highly statistically significant' therefore rest on a test that is either misapplied or misreported. Please provide the actual contingency tables for any statistical test (e.g., a McNemar test for paired data, if appropriate) and correct the claims accordingly.","section":"Section 6, Table 6 (No repetitions)"},{"comment":"The adaptation guidelines instruct informants to rewrite sentences in their dialect, but the paper does not report any verification that the dialectal reformulations preserve the original NLI label (entailment/contradiction/neutral). If a rewrite changes the semantic relation between premise and hypothesis, the gold label becomes noisy, and the observed accuracy drop might reflect label corruption rather than difficulty with dialectal variation. Please either add a label-preservation validation step (e.g., a second annotation or automatic check) or explicitly discuss this as a limitation.","section":"Section 3.1, Appendix A"},{"comment":"For premises with multiple dialectal rewrites, the procedure for selecting which variant is retained in the 'No repetitions' split is not described. This selection can bias the accuracy estimate and hence the comparison with XNLIeunative. Please specify the selection rule and, if feasible, assess robustness by repeating the analysis over different selections or by reporting the full distribution of accuracies across retained variants.","section":"Section 6, Table 6 (construction of No repetitions)"},{"comment":"The claim that encoder-only models 'particularly struggle with Western Basque' is not strongly supported by the reported numbers. In the translate-train setting (Table 13), the lowest accuracy is for Navarrese (71.43), not Western (73.75), and the Western-Central difference is only about 0.7 points. The ablation in Table 6 shows a 0.44-point difference between removing Western and removing Central repeated instances, with no significance test. Please qualify the claim or provide statistical evidence (e.g., a paired test across the retained premise set) that Western examples are significantly harder than Central ones.","section":"Section 5.1, Table 13; Section 7"}],"minor_comments":[{"comment":"Several cells in the PDF rendering appear to run together (e.g., '83.4275.6373.21'); please ensure the table is formatted with clear separation between values.","section":"Table 4"},{"comment":"The dual-axis plots with Levenshtein distance and accuracy on the same figure may be visually misleading; consider normalizing the two quantities or using separate panels to make the relationship easier to assess.","section":"Figure 1"},{"comment":"Minor naming consistency: the paper alternates between 'XNLIes var' and 'XNLIesvar'; please use the latter uniformly.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The discrepancy between the reported p-values and the counts in Table 6 is the central obstacle; it needs to be resolved before the empirical claims can be accepted. The dataset itself is a useful resource, and the paper's scope is appropriate for the journal, but the statistical analysis must be corrected and the label-preservation concern addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The bottom line: this paper's real contribution is the XNLIvar dataset, the first manually curated NLI benchmark with Basque and Spanish geographic variants. That alone justifies reading it. The evaluation is also reasonably thorough: multiple encoder-only models, three training paradigms, several LLM prompts, plus error analysis. The qualitative pattern—accuracy drops when dialectal variants are introduced—shows up consistently across nearly all configurations, which is a meaningful signal.\n\nBut the paper overstates its quantitative case. The chi-square test in Section 6 is not reproducible from the counts in Table 6. For the \"No repetitions\" Basque split, 75.63% vs 71.77% on 621 items gives a chi-square of about 2.4, p≈0.12; for Spanish, 74.61% vs 73.00% gives p≈0.5. Neither approaches p<.001. The paper calls this a chi-square test of independence, but it reads like a misapplied test—possibly a paired McNemar confusion—and the discordant-pair counts are not reported. Since this test is the only significance support for the headline claim, the statistical backing is not there. The consistent direction of the effect across many experiments mitigates this, but the paper should present the results as a descriptive pattern and either fix the test or drop the significance language.\n\nThe second issue is label preservation. The authors adapted standard sentences into dialects but never verify that the NLI relation (entailment/contradiction/neutral) still holds after adaptation. If some rewrites shift meaning, the accuracy drop could reflect annotation noise rather than model sensitivity to variation. This is a known pitfall in paraphrase evaluation; a small re-annotation study would address it.\n\nThe Western Basque claim is also weaker than the abstract suggests. The per-dialect results in Appendix F show Navarrese (the smallest sample) with the lowest accuracy, and Western only marginally below Central. The ablation difference (73.14 vs 72.70) is a 0.44-point gap with no significance test. This is a minor overinterpretation, not a fatal flaw.\n\nWho should read this: anyone building or evaluating models for Basque or dialectal Spanish, and researchers working on variation-inclusive benchmarks. The dataset itself is a solid resource. But the paper as written needs a serious revision—fix the statistics, add label verification, soften the dialect-specific claims. I'd send it to peer review, but I'd expect the reviewers to require those changes.","headline":"XNLIvar is a genuinely useful new dialectal NLI resource for Basque and Spanish, but the paper's chi-square significance claims don't survive contact with its own table.","tokens_in":17439,"tokens_out":3482,"would_cite":true,"duration_ms":31337,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Including geographic dialect variants in NLI causes a statistically significant accuracy drop in language models — most sharply in Basque — and the drop reflects linguistic variation, not lexical overlap.","keywords":["natural language inference","language variation","dialects","Basque","Spanish","cross-lingual transfer","LLM evaluation","XNLIvar"],"falsifier":"Collect a fresh validation set by having a different, larger panel of native speakers, ideally including non-linguists, naturally retell or rewrite the same standard sentences in their own regional speech, then rerun the best XLM-RoBERTa and Gemma/Llama configurations; if accuracy no longer drops relative to the standard set, the reported effect is an artifact of the original annotator pool and adaptation style.","tokens_in":16442,"feed_emoji":"🗣️","tokens_out":9059,"duration_ms":87544,"temperature":0.7,"pith_summary":"This paper sets out to test whether current language models really understand regional language as it is actually spoken, rather than only the standardized written form. To do this, it introduces XNLIvar, a manually curated natural language inference dataset containing Basque sentences in Western, Central, and Navarrese dialects and Spanish sentences in Cuban, Ecuadorian, Andalusian, Tenerife, and Uruguayan variants, all built from the same native-language source sentences. Across encoder-only and generative models, the paper reports that accuracy drops consistently when dialectal variants replace standard-language test sentences, with the sharpest drop in Basque, and a chi-square test on parallel subsets puts the difference at $p < .001$ for both languages. The authors argue the drop is caused by linguistic variation itself: lexical overlap between premise and hypothesis stays the same, while accuracy still falls. A sympathetic reader would care because it gives a concrete, reusable benchmark for measuring whether language technology is inclusive of dialect speakers, and because it suggests standard benchmarks overstate model competence for real regional language use.","feed_headline":"Dialect variants cut AI language accuracy in Basque and Spanish","feed_subtitle":"New dialect-augmented NLI benchmark shows the loss comes from regional variation, not word overlap.","key_machinery":"The load-bearing object is XNLIvar, a parallel NLI evaluation set built by manually adapting the native Basque XNLIeunative sentences into three Basque dialectal variants and five Spanish variants, keeping every other property of the test identical. Its power comes from controlled comparison: the same source sentences appear in standard and variant form, so any accuracy difference is attributable to variation. The analysis machinery has three parts: Levenshtein distance quantifies how far each dialect lies from the standard; a \"No repetitions\" subset removes multiply-adapted sentences so the standard and variant sets are exactly parallel; and a chi-square test of independence measures whether the accuracy difference is statistically significant. An additional ablation removing Western or Central Basque duplicates isolates the influence of specific dialect groups. Together these pieces separate linguistic variation from lexical overlap and from dataset size.","core_discovery":"The paper's central claim is that language models lose ground on natural language inference the moment test inputs shift from a standardized language to authentic geographical variants, and that this loss is a property of linguistic variation rather than a side effect of lexical similarity between premise and hypothesis. Starting from XNLIeunative, a native Basque NLI dataset, the authors created parallel dialectal versions in Basque and Spanish, then evaluated the same models in model-transfer, translate-train, and translate-test settings and with zero-shot, few-shot, Chain-of-Thought, and QA-prompted generative configurations. Accuracy falls on the variant sets in nearly every configuration, and the fall is statistically significant ($p < .001$) once duplicated dialectal instances are removed; for Basque the drop reaches about two to four percentage points for the best discriminative models and about six to ten points for generative models. Error analysis ties the largest Basque loss to the Western dialect, the most distant from the standard form according to Basque dialectology, while Spanish losses concentrate in variants written with non-standard orthography. The authors therefore conclude that current models under-process dialectal variation, and that XNLIvar can serve as a benchmark for measuring and eventually correcting this gap.","pith_inferences":["If the effect survives a broader speaker sample, dialectal variation should become a first-class evaluation axis for NLP benchmarks in any language with geographic variants; the paper only claims the effect for Basque and Spanish, but nothing in the mechanism is language-specific.","Because the Spanish variants were produced by machine-translating Basque source sentences before manual adaptation, the reported Spanish drop may understate sensitivity to variation found in Spanish-native texts; a native-source Spanish control would be a sharper test.","The orthography finding suggests a testable preprocessing fix the paper does not run: normalize non-standard Spanish orthography (for example, turning \"mihma\" into \"misma\") and check whether the accuracy gap closes.","The released speaker metadata may let future work connect model errors to sociolinguistic variables such as age and gender, since the annotator profiles are recorded."],"forward_implications":["Standard-only NLI benchmarks overestimate how well current models understand real regional language use, because the same native sentences in dialect form score consistently lower.","The Western Basque dialect is the hardest of the well-represented Basque variants for encoder-only models, matching its greater linguistic distance from standard Basque.","For Spanish variants, accuracy losses concentrate in adaptations with non-standard orthography, so the bottleneck is orthographic rather than lexical or grammatical.","The statistically significant $p < .001$ difference on the parallel \"No repetitions\" subsets shows the drop is not an artifact of extra duplicated instances."],"supporting_citations":[{"why":"Supplies XNLI, the cross-lingual NLI benchmark and test-setting template that XNLIvar extends to dialectal variants.","marker":"Conneau et al. (2018)"},{"why":"Provides XNLIeu and XNLIeunative, the standard-language Basque datasets used as the source and control for XNLIeuvar.","marker":"Heredia et al. (2024)"},{"why":"Establishes that train-test distribution shifts and translation artifacts distort cross-lingual NLI, motivating the native-versus-variant comparison.","marker":"Artetxe et al. (2020)"},{"why":"Defines MultiNLI, the NLI task and English training source used for model-transfer experiments.","marker":"Williams et al. (2018)"},{"why":"Supplies the dialectological claim that peripheral Basque dialects such as Western are more distant from the standard, which the per-dialect results corroborate.","marker":"Mitxelena (1981)"},{"why":"Provides prior NLP evidence that Biscayan/Western Basque has the greatest historical distance, supporting the Western-dialect difficulty finding.","marker":"Estarrona et al. (2023)"},{"why":"Documents that orthographic variation degrades model performance, used to interpret the Spanish variant results.","marker":"De la Rosa et al. (2024)"},{"why":"Shows that Spanish varieties share many common, non-dialect-specific examples, explaining annotator difficulty and the smaller Spanish variation gap.","marker":"Lopetegui et al. (2025)"}],"fun_headline_variants":["Dialect variants drop NLI accuracy in Basque, Spanish","Western Basque trips NLI models most, says new benchmark","NLI loss on dialects is not about lexical overlap","Basque and Spanish dialects expose NLI model blind spots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dialectal adaptations in XNLIvar are authentic, representative examples of the named Basque and Spanish geographical variants, despite being produced by only 12 Basque and 6 Spanish speakers, all with linguistics or NLP backgrounds.","fun_headline_variants_meta":{"raw":{"variants":["Dialect variants drop NLI accuracy in Basque, Spanish","Western Basque trips NLI models most, says new benchmark","NLI loss on dialects is not about lexical overlap","Basque and Spanish dialects expose NLI model blind spots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000529,"raw_usage":{"total_tokens":2536,"prompt_tokens":914,"completion_tokens":1622,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":1556}},"tokens_in":530,"tokens_out":1622,"duration_ms":11393,"temperature":1.0,"reasoning_tokens":1556,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:39:31.322450+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a fresh validation set by having a different, larger panel of native speakers, ideally including non-linguists, naturally retell or rewrite the same standard sentences in their own regional speech, then rerun the best XLM-RoBERTa and Gemma/Llama configurations; if accuracy no longer drops relative to the standard set, the reported effect is an artifact of the original annotator pool and adaptation style.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides XNLIeu and XNLIeunative, the standard-language Basque datasets used as the source and control for XNLIeuvar."},{"cited_title":"Julio de Urquijo","cited_arxiv_id":null,"evidence_quote":"Supplies the dialectological claim that peripheral Basque dialects such as Western are more distant from the standard, which the per-dialect results corroborate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides prior NLP evidence that Biscayan/Western Basque has the greatest historical distance, supporting the Western-dialect difficulty finding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that orthographic variation degrades model performance, used to interpret the Spanish variant results."}],"review_version":2}