On Reference (In-)Determinacy in Natural Language Inference

Alex Fabrikant; Chaitanya Malaviya; Dan Roth; Hagai Taitelbaum; Senaka Buthpitiya; Sihao Chen; Tal Schuster

REVIEW 4 major objections 5 minor 12 references

On Reference (In-)Determinacy in Natural Language Inference

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that NLI models trained under the reference determinacy assumption — that premise and hypothesis always describe the same context — systematically fail when applied to retrieved evidence that may concern a different…

desk verdict Useful diagnostic benchmark, but the causal story about reference determinacy is overclaimed and the human-disagreement analysis has a table/text mismatch. read the letter →

arxiv 2502.05793 v1 pith:EJIIPE4I submitted 2025-02-09 cs.CL

Sihao Chen , Chaitanya Malaviya , Alex Fabrikant , Hagai Taitelbaum , Tal Schuster , Senaka Buthpitiya , Dan Roth This is my paper · ORCID

classification cs.CL

keywords naturallanguageinferencereferencedeterminacyfactverificationbenchmarkhumandisagreemententailmentcontradictionretrieval-augmented

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the reference determinacy (RD) assumption baked into most NLI dataset creation — that premise and hypothesis always describe the same context — leaves trained models unable to recognize when a retrieved evidence sentence is irrelevant to a claim. To show this, the authors build RefNLI, a benchmark of 1,143 claim-evidence pairs drawn from fact-verification data, labeled with a four-way scheme that separates true entailment/contradiction from neutral and reference-ambiguous cases. Finetuned NLI models and few-shot LLMs both misclassify these pairs, producing over 80% false contradictions and over 50% false entailments. The paper also finds that reference ambiguity explains part of the human disagreement documented in NLI annotation, and that filtering training examples that rely on the RD assumption improves downstream precision.

What carries the argument

The load-bearing object is the reference determinacy (RD) assumption — the convention that premise and hypothesis are interpreted as referring to the same context when annotators assign an NLI label. The paper's diagnostic instrument is RefNLI, a four-way labeled benchmark (entailment, contradiction, neutral, ambiguous) built by sampling claims from FEVER and VitaminC and retrieving Wikipedia evidence with BM25. The ambiguous label is the key addition: it marks cases where the premise could support or contradict the claim under some resolution of its references, so the correct three-way label cannot be determined without the RD assumption. The machinery works by measuring how often models trained under RD assign entailment or contradiction to such pairs.

What would settle it

Train a model on an NLI dataset re-annotated without the RD assumption (or on ANLI, where RD is least enforced) and evaluate on RefNLI; if false contradiction and entailment rates remain above 80% and 50%, the RD-assumption causal story is not supported. A more targeted test would vary the Jaccard filtering threshold in the mitigation experiment and show that the precision gain disappears at thresholds not chosen by the authors.

Watch

Extended reading notes

Core claim

The central discovery is that current NLI systems carry a reference determinacy bias: because they are finetuned exclusively on premise-hypothesis pairs annotated under the assumption that both texts refer to the same context, they systematically over-predict entailment and contradiction when applied to inputs where the premise is retrieved evidence that may concern a different entity or event. On the RefNLI benchmark, every training-mixture tested shows low precision on contradiction and entailment predictions, with the majority of true cases being neutral. The bias appears across finetuned T5 models and in few-shot prompted Gemini, and it transfers to fact verification: models predict contradictions against irrelevant Wikipedia pages far more often than entailments. A simple lexical-overlap filter on training data reduces the bias, and the paper shows that human raters also disagree more on examples with reference ambiguity, linking the RD assumption to known annotation disagreements.

Load-bearing premise

The paper assumes that the reference determinacy assumption in training data, rather than other differences between datasets such as annotation style, domain, or label balance, is what causes the observed false contradiction and entailment behavior.

Editorial extensions

If this is right

NLI models used in retrieval-based fact verification will keep producing false contradictions against irrelevant evidence unless trained or prompted to check reference determinacy.
Filtering low-lexical-overlap contradiction and entailment examples from training data improves precision on RefNLI, so dataset curation can partially mitigate the bias.
The four-way label set including 'ambiguous' gives a practical protocol for flagging claims that should not be verified from retrieved evidence.
Human disagreement on NLI labels, especially contradiction versus neutral, is partly explained by reference ambiguity rather than purely subjective judgment.
Any downstream task that feeds retrieved text into an NLI model inherits the reference determinacy bias.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that the bias likely extends beyond fact verification to any retrieval-augmented pipeline that uses NLI-style judgments, such as hallucination detection and text attribution, since those also pass retrieved snippets as premises.
A design fix the paper does not explore is adding an explicit relevance-checking head or an 'ambiguous/neutral' output category to NLI models trained for downstream use, rather than relying on lexical filtering.
The observed correlation between ambiguity and human disagreement suggests that disagreement-aware training, which models label distributions rather than single majority labels, may partly absorb the RD effect.
RefNLI could be extended to other languages and to multi-sentence evidence to test whether the bias scales with context length.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this paper builds a small, carefully annotated benchmark (RefNLI) that shows NLI models, both fine-tuned and few-shot LLMs, frequently call 'contradiction' on evidence that is simply about a different context. That result is real and worth having. The 4-way label set (entailment/contradiction/neutral/ambiguous) is a good idea, and the annotation protocol is serious: 1,143 pairs, two-author adjudication, and 0.83 Fleiss kappa on a subsample.

What it does well: it makes a concrete, reproducible artifact. The GitHub link and the detailed sampling procedure mean others can build on this. The observation that models produce many false contradictions on irrelevant Wikipedia pages (Figure 1) is a clean demonstration of a known but underspecified problem.

Now the soft spots. The causal narrative — that the reference determinacy (RD) assumption in training data causes the bias — is not established. Section 4.2 compares models trained on different dataset mixtures and finds ANLI best, but the authors themselves concede confounders. The only direct manipulation is the Jaccard filter in Section 4.4, which is a proxy for lexical overlap, not reference determinacy. Low-overlap pairs can be perfectly deterministic; high-overlap pairs can be ambiguous. The filter also rebalances labels, so the precision gains could come from removing hard examples or changing class priors. Without validating the heuristic against human RD judgments, it doesn't carry the causal weight the paper puts on it.

There is also an internal inconsistency in Section 5. Table 6 shows that on both SNLI and MNLI, the contradiction-vs-neutral correlation is more negative (i.e., raters distinguish them better) for ambiguous examples than for unambiguous ones (-0.61 vs -0.23 on SNLI; -0.25 vs -0.03 on MNLI). That is the opposite of what the text claims. The paper says agreement improves when going from ambiguous to unambiguous, but the numbers say the reverse. Either the interpretation of the correlation sign is wrong or the rows are mislabeled. This needs fixing.

Minor issues: the benchmark is sampled from model-positive predictions on 'irrelevant' pages, so it is a diagnostic, not an estimate of error rates in the wild. The contradiction and entailment cells are small (66 and 37), so percentages come with wide intervals. The paper also has a couple of typos, including a duplicated 'SNLI' where MNLI is meant.

Who is this for: researchers working on NLI, fact verification, and annotation artifacts. The benchmark itself is a solid contribution, and the mitigation experiment, even with its flaws, is a reasonable starting point. It deserves a serious referee, but the authors should be pushed to either soften the causal claims or validate the filtering heuristic.

Referee Report

4 major / 5 minor

Summary. The paper revisits the reference determinacy (RD) assumption in NLI dataset creation and argues that models trained under this assumption fail when the premise and hypothesis refer to different contexts. The authors introduce RefNLI, a benchmark of 1,143 claim-evidence pairs sampled from FEVER and VitaminC retrieval settings, labeled by expert annotators as entailment, contradiction, neutral, or ambiguous. They evaluate finetuned T5-large models trained on different NLI dataset mixtures, plus Gemini 1.0 Ultra with few-shot prompting, and report many false contradiction and entailment predictions. They also propose a Jaccard-overlap filtering heuristic to mitigate the RD bias and present a ChaosNLI-based analysis linking reference ambiguity to human disagreement patterns.

Significance. The central observation that sentence-pair NLI models produce confident entailment and contradiction predictions on irrelevant retrieved evidence is practically important for fact verification and other downstream applications. The paper contributes a new diagnostic benchmark, reports a reasonable inter-annotator agreement (0.83 Fleiss kappa) on a subsample, evaluates multiple training-dataset mixtures, and includes candid limitation statements. The benchmark itself is a useful resource. However, the significance of the paper depends on whether the evaluation protocol treats the Ambiguous class correctly, whether the sampling procedure supports the headline false-prediction rates, and whether the causal attribution to the RD assumption is supported by the experiments; in the current version those load-bearing points need further work.

major comments (4)

[Section 3.2 / Table 3] The benchmark construction introduces a selection bias that is not accounted for in the reported statistics. The annotation sample was drawn only from pairs where the RoBERTa NLI model predicted entailment or contradiction and where the evidence did not come from the correct Wikipedia page, so the benchmark is conditional on model-positive predictions on irrelevant pages. The abstract's statement that models lead to '>80% false contradiction and >50% entailment predictions' should be framed as behavior on this filtered set, not on randomly retrieved evidence. Please report the size of the sampling pool, the distribution of model predictions on the full retrieved set, and present the RefNLI results with this conditioning made explicit.
[Section 4.1 / Table 3] The evaluation protocol does not state how the 135 Ambiguous examples in RefNLI are treated. Table 3 reports metrics only for Contradiction, Neutral, and Entailment, and the label counts in the table sum to 1,008, not 1,143. If ambiguous examples are excluded from all metrics, the reported precision and recall numbers are computed on a subset of the benchmark and the headline false-prediction rates are not directly comparable to a 3-way NLI evaluation; if they are mapped to another label, that mapping should be justified. The paper should report a full confusion matrix over all four labels and clarify the exact evaluation denominator.
[Section 4.2 / Section 4.4] The causal attribution from the RD assumption in training data to the observed model behavior is not established. Section 4.2 compares models trained on different dataset mixtures and notes that ANLI, where RD is least enforced, is most useful for contradiction detection, but the paper concedes that 'no definitive conclusion can ever be drawn here due to the existence of many other confounders.' The only direct manipulation, the Jaccard filter in Section 4.4, uses a threshold of 0.15 that is never validated against human RD judgments, and the experimental protocol also rebalances label proportions, so the precision improvements in Table 5 could result from removing hard examples, changing class priors, or reducing label noise rather than from removing RD-dependent examples. A matched control condition (e.g., random filtering of the same size, or a threshold sweep calibrated on human-annotated ambiguity) is needed before the abstract and conclusion can claim a causal role for the RD assumption.
[Section 5 / Table 6] The interpretation of the correlation analysis appears inconsistent with the reported numbers. The paper states that a correlation of -1 indicates perfect agreement among raters on the distinction between two labels, so a more negative correlation means less confounding between labels. With that definition, the SNLI contradiction-neutral correlation moves from -0.61 for ambiguous examples to -0.23 for unambiguous examples, meaning raters actually confound contradiction and neutral more, not less, when the reference is unambiguous. The text claims the opposite ('the rater agreement between contradiction and neutral improves when we go from ambiguous to unambiguous cases'). This inversion undermines the human-disagreement contribution as currently written and requires re-analysis and correction.

minor comments (5)

[Throughout] There are several typos: 'parathenses' in the Table 3 caption, 'discrepency' in Section 4.2, 'applciations' in Section 6, and 'the the effect' in the contributions list in Section 1.
[Section 5] The sentence '˜ 47%in SNLI, ˜ 46%in SNLI' appears to contain a typo and should read '˜47% in SNLI, ˜46% in MNLI'.
[Section 3.1 / Figure 1] The figure would be more informative if it reported the number of predictions in each bar and the number of retrieved sentences per claim; currently the reader cannot assess whether the contradiction-heavy distribution is driven by a small number of claims.
[Section 4.1] The AUROC computation is described only as 'one-label-vs-rest'; please specify the exact probability aggregation over the T5 output space and how ties or low-confidence predictions are handled.
[Section 3.2] The annotation protocol says one author produced the initial label and another verified it, with a separate 102-example subsample used for the kappa computation; please clarify whether the kappa reflects agreement among three authors on the same 102 examples, which is how it appears in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core evaluation uses a human-labeled benchmark and external datasets; no prediction reduces to its input by construction.

full rationale

This is an empirical paper rather than a formal derivation, and its central claims are tested against a newly constructed human-annotated benchmark, RefNLI. Although candidate pairs for RefNLI were pre-filtered using a RoBERTa NLI model ('From the set of examples where the RoBERTa NLI model predicts entailment or contradictions... we sample a subset for human annotation uniformly at random', Section 3.2), the benchmark labels themselves were produced by expert human annotation under a fixed protocol and are independent of the models later evaluated. The evaluated T5, Gemini, and related models are not fitted to those labels, so their measured false contradiction and entailment rates are genuine empirical outcomes rather than construction artifacts. The Jaccard-similarity filter in Section 4.4 is a heuristic training-data preprocessing choice (threshold 0.15), not a parameter fitted to RefNLI, so the mitigation experiment does not feed the test labels back into the training procedure. The paper's causal attribution of the observed bias to the reference-determinacy assumption is admittedly confounded ('no definitive conclusion can ever be drawn here due to the existence of many other confounders', Section 4.2), but confounding is a validity limitation, not circularity. Self-citations to prior datasets such as VitaminC are not load-bearing in a circular sense, since those datasets are public external resources and no author-specific uniqueness theorem is invoked to force the paper's conclusions. No equation or fitted quantity is reused as a predicted value by definition, so no circular step can be exhibited; the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper does not derive a mathematical result; it makes empirical claims. The main axiomatic choices are the 4-way label set for RefNLI, the use of BM25-retrieved Wikipedia evidence as a proxy for fact verification, the interpretation of ChaosNLI labels as reflecting RD, and the assumption that human annotation of reference ambiguity is reliable. The only hand-set numerical free parameter is the Jaccard threshold in the filtering experiment.

free parameters (1)

Jaccard similarity threshold = 0.15
Hand-chosen in Section 4.4 to discard training examples whose premise and hypothesis share low lexical overlap, conjectured to be RD-dependent. Not tuned by validation.

assumptions (4)

domain assumption The four-way label set (Entailment, Contradiction, Neutral, Ambiguous) is the appropriate decomposition for fact verification judgments.
Introduced in Section 3.2; the paper argues that the Neutral vs Ambiguous distinction depends on whether any reference assignment could make the evidence relevant.
domain assumption BM25 retrieval from a 2018 English Wikipedia dump, with evidence sentences from pages other than the claim's correct page, is a good proxy for realistic fact verification input.
Used in Section 3.1 to construct RefNLI; the paper does not validate this proxy against other retriever and evidence distributions.
domain assumption Reference ambiguity labels assigned by the authors are reliable, based on a 102-example subsample with Fleiss' kappa 0.83.
Section 3.2 reports inter-rater agreement on a small subsample; the full 1,143 labels are produced by one author and verified by another.
domain assumption The original SNLI and MNLI labels assume reference determinacy, while ChaosNLI annotations do not impose this assumption.
This is the premise of Section 5's analysis of human disagreement; it rests on the dataset descriptions cited in the paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On Reference (In-)Determinacy in Natural Language Inference." pith.science (2026). https://pith.science/paper/EJIIPE4I

@misc{pith2026250205793,
  author       = {Pith},
  title        = {Pith review of: On Reference (In-)Determinacy in Natural Language Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJIIPE4I}},
  note         = {Machine review of arXiv:2502.05793}
}

read the original abstract

We revisit the reference determinacy (RD) assumption in the task of natural language inference (NLI), i.e., the premise and hypothesis are assumed to refer to the same context when human raters annotate a label. While RD is a practical assumption for constructing a new NLI dataset, we observe that current NLI models, which are typically trained solely on hypothesis-premise pairs created with the RD assumption, fail in downstream applications such as fact verification, where the input premise and hypothesis may refer to different contexts. To highlight the impact of this phenomenon in real-world use cases, we introduce RefNLI, a diagnostic benchmark for identifying reference ambiguity in NLI examples. In RefNLI, the premise is retrieved from a knowledge source (i.e., Wikipedia) and does not necessarily refer to the same context as the hypothesis. With RefNLI, we demonstrate that finetuned NLI models and few-shot prompted LLMs both fail to recognize context mismatch, leading to over 80% false contradiction and over 50% entailment predictions. We discover that the existence of reference ambiguity in NLI examples can in part explain the inherent human disagreements in NLI and provide insight into how the RD assumption impacts the NLI dataset creation process.

Figures

Figures reproduced from arXiv: 2502.05793 by the authors.

**Figure 2.** Distribution of the majority labels from the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Confusion Matrices between majority label from the original annotation vs. ChaosNLI’s reannotation label for SNLI and MNLI examples from Nie et al. (2020b). Label: Skip, since it’s unclear in the hypothesis what “based off of a poem” means B Human Disagreements and Reference Ambiguity [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 10 canonical work pages

[1]

Skip unclear claims or premises : If you think the claim is difficult to understand, or there is too much ambiguity, skip the claim entirely

work page
[2]

The sky is blue

Don’t label the claim by its truth value in the world: If a claim says “The sky is blue”, and the premise says something completely different, label it as neutral. Don’t label such cases as entailment based on just your world knowledge

work page
[3]

If needed, a web search is al- lowed when making the judgements

World Knowledge is permitted: You can as- sume commonly accepted world knowledge when interpreting the premise, e.g., basic ge- ography and other commonsense knowledge are allowed. If needed, a web search is al- lowed when making the judgements. How- ever, don’t make too many inferences

work page
[4]

If the premise clearly indicates a time of an event, but the claim doesn’t, assume that the claim is uttered right after the event

Temporal considerations: Ignore tense (e.g., past or present) in both the premise and claims. If the premise clearly indicates a time of an event, but the claim doesn’t, assume that the claim is uttered right after the event

work page
[5]

A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguis- tics. Jingjing Xu, Y...

work page 2018
[6]

The Hunchback of Notre Dame

Neutral vs. Ambiguous Reference: The dis- tinction between the two can be difficult some- times. The general rule is: if the premise can’t seem to support the claim no matter how you interpret the premise, then it’s neutral. Some examples given in the instructions Premise: Wales has a large region rich in coal deposits. Hypothesis: The Ural Mountains cont...

work page 2020
[11]

ambiguous

Personal surnames: If only the surname of a person is mentioned in the premise, and there’s not enough evidence for in the premise for you to determine the last name is referring to the same entity as in the hypothesis, mark the example as “ambiguous”

work page
[2005]

In Machine learning challenges workshop, pages 177–190

The PASCAL recognising textual entailment challenge. In Machine learning challenges workshop, pages 177–190. Springer. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Associa...

work page 2019

Show all 12 references

[2018]

Annotation artifacts in natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Assoc...

2018 arXiv
[2019]

Asking clarification questions in knowledge- based question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1618–1629, Hong Ko...

2019 arXiv
[2021]

In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pages 2356–2362

Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pages 2356–2362. Alisa Liu,...

2021 arXiv
[2023]

arXiv preprint arXiv:2312.11805

Gemini: a family of highly capable multi- modal models. arXiv preprint arXiv:2312.11805. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith

arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Skip unclear claims or premises : If you think the claim is difficult to understand, or there is too much ambiguity, skip the claim entirely

work page

[2] [2]

The sky is blue

Don’t label the claim by its truth value in the world: If a claim says “The sky is blue”, and the premise says something completely different, label it as neutral. Don’t label such cases as entailment based on just your world knowledge

work page

[3] [3]

If needed, a web search is al- lowed when making the judgements

World Knowledge is permitted: You can as- sume commonly accepted world knowledge when interpreting the premise, e.g., basic ge- ography and other commonsense knowledge are allowed. If needed, a web search is al- lowed when making the judgements. How- ever, don’t make too many inferences

work page

[4] [4]

If the premise clearly indicates a time of an event, but the claim doesn’t, assume that the claim is uttered right after the event

Temporal considerations: Ignore tense (e.g., past or present) in both the premise and claims. If the premise clearly indicates a time of an event, but the claim doesn’t, assume that the claim is uttered right after the event

work page

[5] [5]

A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguis- tics. Jingjing Xu, Y...

work page 2018

[6] [6]

The Hunchback of Notre Dame

Neutral vs. Ambiguous Reference: The dis- tinction between the two can be difficult some- times. The general rule is: if the premise can’t seem to support the claim no matter how you interpret the premise, then it’s neutral. Some examples given in the instructions Premise: Wales has a large region rich in coal deposits. Hypothesis: The Ural Mountains cont...

work page 2020

[7] [11]

ambiguous

Personal surnames: If only the surname of a person is mentioned in the premise, and there’s not enough evidence for in the premise for you to determine the last name is referring to the same entity as in the hypothesis, mark the example as “ambiguous”

work page

[8] [2005]

In Machine learning challenges workshop, pages 177–190

The PASCAL recognising textual entailment challenge. In Machine learning challenges workshop, pages 177–190. Springer. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Associa...

work page 2019