Pith. sign in

REVIEW 3 major objections 6 minor 52 references

NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models can score high on natural language inference while disagreeing with their own judgments on the atomic facts that compose the hypothesis, so accuracy alone overstates how well they reason.

desk verdict Full-example accuracy overstates LLM consistency on atomic sub-problems; the paper's new IC metric, however, is estimated under an independence assumption that reduces it to a function of per-bucket accuracy and should be reworked before the metric is used. read the letter →

arxiv 2502.08080 v2 pith:BXRKWMGI submitted 2025-02-12 cs.CL

Neha Srikanth , Rachel Rudinger This is my paper · ORCID
classification cs.CL
keywords atomichypothesisdecompositionnaturallanguageinferencedefeasiblelogicalconsistencylargemodelscriticalatomsinferentialδ-NLI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that high accuracy on natural language inference (NLI) benchmarks does not mean a model reasons consistently. The authors decompose hypotheses into atomic propositions and form atomic sub-problems—individual judgments about whether a premise entails each granular fact—then check each model's full-example label against its own atomic judgments. Across six large language models on SNLI, they find substantial logical inconsistency, and the same pattern appears in defeasible NLI, where an update strengthens or weakens an inference. They also introduce critical atoms, the fact an update most directly targets, and an inferential consistency metric that measures whether a model is consistently right or wrong about the same fact across different contexts. The reason to care is that accuracy alone appears to overstate how well models understand the situations they are reasoning about.

What carries the argument

The central machinery is atomic hypothesis decomposition paired with two derived instruments. Hypotheses are divided into atomic propositions using exemplar-prompted LLM generation, so each atom is a granular fact the hypothesis entails; pairing the premise with each atom creates an atomic sub-problem. In traditional NLI, consistency is checked through logical rules that relate sub-problem labels to the full label. In defeasible NLI, the paper defines the critical atom as the atom whose label most strongly matches the overall update polarity, which identifies the question under discussion (QUD) the example targets. The inferential consistency metric (IC) groups examples sharing a critical atom into buckets and estimates, from each bucket's average correctness $\theta$, the probability that two examples in the bucket are both correct or both incorrect: $IC = \mathbb{E}[\theta^2] + \mathbb{E}[(1-\theta)^2]$. This is meant to capture whether a model is consistently right or wrong about the same fact under different contexts.

What would settle it

Compute the empirical pairwise agreement over all pairs of δ-NLI examples that share a critical atom and compare it with the value predicted by equation (2); a systematic gap for models with stable fact-level errors would show that the independence assumption fails and the metric misstates consistency.

Watch

Extended reading notes

Core claim

The central discovery is that a model's full-example NLI prediction and its predictions on the atomic sub-problems that compose the hypothesis frequently disagree, in ways accuracy numbers hide. On a sample of SNLI, the paper measures overall logical consistency with three rules: if the full pair is entailed, every valid atom must be entailed; if contradictory, at least one atom must contradict; if neutral, at least one atom must be neutral and none may contradict. Every model tested falls below its full-example accuracy on this measure, and consistency gaps are largest on examples the model got wrong. In the defeasible setting, the authors identify critical atoms—the atom(s) an update most strongly affects—and show that models handle those better than non-critical atoms, yet still show inconsistency. Their headline metric, inferential consistency (IC), is the probability that two δ-NLI examples sharing a critical atom are both predicted correctly or both incorrectly; by that measure, the top-scoring model reaches 88.7, and even it is not consistent about facts such as "the people are tall" across different contexts.

Load-bearing premise

The headline consistency measure assumes that a model's correctness on two examples that share a critical atom behaves like two independent coin flips with the same success rate; if a model is stably right or stably wrong about that fact, the reported consistency number will be off.

Editorial extensions

If this is right

  • Accuracy on full NLI examples is not a reliable proxy for reasoning consistency: a model can be consistently wrong about the same fact across contexts while still scoring high overall.
  • Logical consistency checks on atomic sub-problems give model developers a finer-grained signal for where reasoning breaks down, such as hypernyms, rare syntactic constructions, or weak effects that compound only in full hypotheses.
  • In defeasible NLI, correctly solving the critical atom is a strong predictor of full-example correctness, while several models still get full examples right after missing the critical atom, suggesting shortcuts unrelated to the targeted inference.
  • The δ-SNLI test set's 1,761 examples collapse to only 349 unique critical atoms (roughly 223 unique questions under discussion), so apparent benchmark diversity overstates the variety of inferences actually being tested.
  • The inferential consistency metric offers a complementary model-comparison axis that ranks models differently from full accuracy and can inform data collection toward underrepresented critical atoms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If atomic consistency reflects genuine understanding, the same decomposition check could be applied to fact-checking and claim-verification pipelines, flagging systems that are accurate on whole claims but inconsistent on the underlying facts even without a new benchmark.
  • The IC estimate assumes independence across examples sharing a critical atom; measuring pairwise consistency directly, or incorporating correlation, would likely change model rankings and is a natural robustness test of the headline metric.
  • Because atoms are generated by an LLM and human-validated only for δ-NLI, the SNLI consistency numbers inherit whatever coverage or bias the generator has; a different decomposition model could shift the measured consistency gaps.
  • A direct testable extension is to fine-tune a model on atomic sub-problems with the full five-point scale (including "no effect") and check whether improved atom accuracy also lifts full-example consistency rather than just full-example accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper decomposes hypotheses in SNLI and δ-SNLI into atomic propositions and evaluates six LLMs on the resulting atomic sub-problems. It reports that LLMs achieve high full-example accuracy but are often logically inconsistent with their own atomic judgments. It introduces a 'critical atom' notion to identify the inference a defeasible example mainly targets, and proposes an 'inferential consistency' metric that measures whether models make consistently correct or incorrect predictions on examples sharing a critical atom. The paper also analyzes dataset diversity through critical-atom grouping and connects critical atoms to questions under discussion.

Significance. If the results hold, the paper offers a useful diagnostic tool for inspecting model reasoning and benchmark content. The careful human validation of δ-SNLI atoms, the release of code and data, and the proposal of a new consistency metric are strengths. However, the headline IC metric rests on an untested independence assumption, the critical-atom analysis has a selection-circularity concern, and the SNLI consistency measurement is model-dependent. These issues affect the interpretation of the main quantitative claims, though they are local and fixable.

major comments (3)
  1. [Section 6, Eq. (2)] The estimator in Eq. (2), IC = E[θ²] + E[(1−θ)²], is exact only when the correctness indicators of two examples within a critical-atom bucket are independent draws from a distribution with mean θ. The paper neither states nor tests this assumption. If, as the paper's own Figure 4 suggests, a model can have a stable misconception about a fact (e.g., always wrong about 'The people are friends'), the two indicators are positively correlated, and the true pairwise probability in Eq. (1) is larger than the reported estimate by the term 2·E[ρ·θ·(1−θ)]. Consequently, the IC values in Table 4 are systematically biased, and model rankings could change if within-bucket correlation varies across models. I recommend computing IC directly from pairwise comparisons within each bucket (the fraction of pairs that are both correct or both incorrect) rather than from mean accuracies, or else explicitly modeling the correlation.
  2. [Section 5.2, Table 3] Critical atoms are selected as the valid atoms whose gold labels have the strongest polarity matching the overall example's gold label. This makes the high values of P(Full✓ | Critical✓) in Table 3 partly a selection artifact: the critical atom is by construction the atom most aligned with the full label, so a model that predicts it correctly has already identified the main evidence for that label. To support the interpretive claim that critical atoms are the 'primary' fact tested by an example, the paper should report a baseline, e.g., the same conditional probabilities for non-critical atoms matched on label strength, or for randomly selected atoms. Without such a baseline, the analysis in Table 3 does not uniquely support the critical-atom story.
  3. [Section 4.1 and §3.2] In the SNLI experiments, the consistency rules are applied only to the atoms that each model itself predicts as entailed by H. Because models differ in which atoms they accept, the consistency rates in Table 2 are not directly comparable across models; a model that is more conservative in accepting atoms may avoid some consistency checks (e.g., it can reject a contradictory atom and thus evade the contradiction rule). The completeness checks in Appendix A.1 address generation coverage, but not the model-dependent filtering. I recommend measuring consistency on a fixed atom set (e.g., the generated atoms without model filtering, or a gold-validated subset) or reporting a sensitivity analysis across alternative atom sets.
minor comments (6)
  1. [Table 1] In row 3, two atoms are both labeled a6; the numbering should be sequential.
  2. [Figure 2] The caption says '1,761 δ-SNLI instances' while Table 5 reports 1,837 test examples; the caption should note that the figure covers only examples with at least one valid atom.
  3. [Equation (2)] The notation uses θ both as the mean correctness of a bucket and as a random variable inside the expectation; please distinguish the bucket-specific mean, e.g., θ_b.
  4. [Table 3] The column header 'P(Full✓| Critical✓)' is ambiguous: the prose says 'correctly ... solved all critical atomic sub-problems', so the header should indicate that the condition is 'all critical atoms correct' rather than 'critical atom correct'.
  5. [Section 5.1] In the second paragraph, 'Since updates often act on multiple parts U' should presumably read 'multiple parts of U' or 'multiple atoms'.
  6. [Section 6] The threshold θ = 0.75 for semantic equivalence is ad hoc; a sensitivity analysis would strengthen the robustness of the bucketing.

Circularity Check

1 steps flagged · score 6.0 of 10

Critical-atom selection is tied to the full label by construction, making the reported conditional accuracy partly self-fulfilling; the paper's central logical-consistency finding remains independent.

  1. self definitional [Section 5.2, 'Identifying Critical Atoms of Updates' and 'Performance on Critical Atomic Sub-Problems versus Full Examples']
    "In order to identify the critical atom for a defeasible NLI example, we identify the subset of its valid atoms with the strongest labels that match its overall polarity. ... We also measure the probability that a model correctly predicts the label for the full example given that it has correctly (Column 4) and incorrectly (Column 5) solved all critical atomic sub-problems (Table 3). Correctly solving all atomic sub-problems is a strong indicator that a model is likely to predict the full problem correctly."

    Critical atoms are defined as the atoms whose gold labels most strongly match the gold full-example label. Therefore, a model that correctly predicts the critical atom has, by construction, gotten the sub-problem that was selected for its alignment with the overall label; P(Full correct | Critical correct) is inflated because the two events share the same gold polarity by selection. Conversely, an error on the critical atom removes the strongest piece of label-aligned evidence, making full-example errors more likely. The paper presents this conditional as an empirical indicator of model reasoning, but the correlation is in part a selection artifact of the critical-atom definition rather than an independent discovery.

full rationale

The paper's headline finding—that LLMs have high full-example accuracy but low logical consistency on atomic sub-problems—is not circular: it follows from direct evaluations of SNLI and delta-SNLI sub-problems against logical consistency rules and manual annotations, and it is checked against a completeness analysis. The IC metric in Eq. 2 is an estimated transform of bucket-level accuracy rather than a fitted prediction; its independence assumption is a validity concern, not a circularity. The clearest circular step is the critical-atom analysis in Section 5.2: critical atoms are selected as the atoms whose gold labels best match the overall gold polarity, and the paper then reports that correctness on critical atoms strongly predicts correctness on the full example. That correlation is partly definitional, since the selection already encodes the label it is used to predict. The QUD framing and atom-bucket diversity analysis do not add further circularity. There is no load-bearing self-citation chain or imported uniqueness theorem. Overall, the central consistency claim is independent, but one supporting result reduces partially to its own selection criterion, warranting a partial-circularity score of 6.

Assumptions & free parameters 1 free parameters · 5 assumptions · 2 invented entities

The analysis rests on author-generated atoms, manual labels, an arbitrary similarity threshold, and an independence assumption in the IC estimator. These are the main inputs the reader pays for rather than receives from established benchmarks. The completeness analysis shows the atom-generation assumption fails in a small number of cases, and the IC estimator is not a direct measurement of pairwise agreement.

free parameters (1)
  • cosine similarity threshold theta = 0.75
    Hand-chosen threshold for connecting critical atoms into semantic-equivalence cliques; no sensitivity analysis is reported, and the resulting 349 unique critical atoms and IC buckets depend on it. Section 6.
assumptions (5)
  • domain assumption Valid atomic decompositions are logically entailed by the hypothesis and cover all of its information.
    Invoked in Sections 3.2 and 4.1; the logical consistency rules require that atoms are entailed by H and that no important atom is missing. The paper's completeness checks show missing atoms occur in a small fraction of examples (Tables 7 and 8), so this is an approximation.
  • domain assumption The consistency rules in Section 4.1 correctly relate atomic sub-problem labels to overall NLI labels.
    Rules 1 to 3 define what counts as consistency. They rely on the atom set being complete and on the semantics of entailment, contradiction, and neutral; the neutral rule is a modeling choice about how to weigh atoms.
  • ad hoc to paper Correctness on two delta-SNLI examples that share a critical atom is independent and identically distributed for the IC estimator.
    Section 6, Eq. 2 estimates IC as E[theta^2] + E[(1-theta)^2]. This is only a probability of matching outcomes under independence; the paper does not test or justify this assumption.
  • ad hoc to paper Semantic equivalence of critical atoms can be decided by an embedding similarity threshold and bidirectional NLI entailment.
    Section 6 uses cosine similarity above 0.75 plus a DeBERTa NLI model to build cliques of equivalent atoms. The threshold is arbitrary and no sensitivity analysis is reported.
  • domain assumption Each update in delta-SNLI targets a critical atom corresponding to a question under discussion.
    Section 5.2 builds the QUD interpretation on this premise. Figure 7 shows 52% of updates target all valid atoms at once, which complicates the single-focus assumption.
invented entities (2)
  • critical atom
    purpose: Designates the atom or atoms an update most strongly affects, used to identify the question under discussion and to group delta-SNLI examples for the inferential consistency metric.
    Critical atoms are an analytical construct defined by the authors using manual labels. The paper provides no falsifiable handle outside its own annotation pipeline; whether a model gets a critical atom is a derived quantity, not an independently measurable entity.
  • inferential consistency (IC)
    purpose: Metric intended to capture the probability that a model is consistently correct or incorrect about the same fact under different contexts.
    IC is a proposed summary statistic computed from bucket accuracies. It has no external benchmark or predicted observable; it is a definitional metric rather than an independently testable entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals." pith.science (2026). https://pith.science/paper/BXRKWMGI

@misc{pith2026250208080,
  author       = {Pith},
  title        = {Pith review of: NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BXRKWMGI}},
  note         = {Machine review of arXiv:2502.08080}
}
read the original abstract

Decomposition of text into atomic propositions is a flexible framework allowing for the closer inspection of input and output text. We use atomic decomposition of hypotheses in two natural language reasoning tasks, traditional NLI and defeasible NLI, to form atomic sub-problems, or granular inferences that models must weigh when solving the overall problem. These atomic sub-problems serve as a tool to further understand the structure of both NLI and defeasible reasoning, probe a model's consistency and understanding of different inferences, and measure the diversity of examples in benchmark datasets. Our results indicate that LLMs still struggle with logical consistency on atomic NLI and defeasible NLI sub-problems. Lastly, we identify critical atomic sub-problems of defeasible NLI examples, or those that most contribute to the overall label, and propose a method to measure the inferential consistency of a model, a metric designed to capture the degree to which a model makes consistently correct or incorrect predictions about the same fact under different contexts.

Figures

Figures reproduced from arXiv: 2502.08080 by the authors.

Figure 1
Figure 1. Top: Atomic hypothesis decomposition breaks down hypotheses (H) into entailed propositional “atoms” (a1−a3). Middle: Pairing the premise (P) with each atom yields a set of NLI sub-problems (P + a); the sub-problem labels predict the full NLI problem (P +H) label. Bottom: Paired with an update (U), each atom yields a defeasible NLI sub-problem (P + a + U); the set of sub-problem labels are predictive of the full prob… view at source ↗
Figure 2
Figure 2. A rug plot visualization of 1,761 δ-SNLI instances and their corresponding distribution of atomic sub￾problem labels. Each vertical slice represents one full δ-SNLI instance. Slice color (red or green) represents the full instance label (weakener or strengthener). For each δ-SNLI problem, we manually label each corresponding atomic sub-problem on a -2 (strongly weakens) to +2 (strongly strengthens) scale. Each verti… view at source ↗
Figure 3
Figure 3. Updates (U) may act on the same hypothesis H in different ways by targeting different atoms. Here, each U strongly targets a different atom, while having no effect on the other atoms derived from H (e.g. the U in the first row has no effect on a3 in the last row). We refer to the atom(s) which an update most strongly affects as the “critical” atom of the (P, H, U) δ-NLI ex￾ample. Critical atoms help identify the que… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Grouping examples by their critical atom(s) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Distribution of fine-grained labels across all [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 7
Figure 7. Figure 7: Proportion of valid atoms used as critical [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 22 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Anton Benz and Katja Jasinskaja. 2017. Questions under discussion: From sentence to discourse

    2017

  4. [4]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. https://doi.org/10.18653/v1/D15-1075 A large annotated corpus for learning natural language inference . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632--642, Lisbon, Portugal. Association for Computational Linguistics

    doi:10.18653/v1/d15-1075 2015

  5. [5]

    H \'e ctor-Neri Casta \ n eda. 1967. Comments on d. davidson's `the logical form of action sentences'

    work page 1967

  6. [6]

    Juan Luis Castro, Enric Trillas, and Jose Manuel Zurita. 1998. Non-monotonic fuzzy reasoning. Fuzzy Sets and Systems, 94(2):217--225

    work page 1998

  7. [7]

    Angelica Chen, Jason Phang, Alicia Parrish, Vishakh Padmakumar, Chen Zhao, Samuel R Bowman, and Kyunghyun Cho. 2023. Two failures of self-consistency in the multi-step reasoning of llms. arXiv preprint arXiv:2305.14279

    arXiv 2023

  8. [8]

    Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. 2024. https://doi.org/10.18653/v1/2024.naacl-long.196 Complex claim verification with evidence retrieved in the wild . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Paper...

    doi:10.18653/v1/2024.naacl-long.196 2024

Show all 52 references
  1. [9]

    Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. 2023. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation. arXiv preprint arXiv:2310.18235

    2023 arXiv

  2. [10]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37--46

    1960

  3. [11]

    David Dowty. 1991. Thematic proto-roles and argument selection. language, 67(3):547--619

    1991

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. arXiv preprint arXiv:2011.00620

    2020 arXiv

  6. [14]

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1--9

    2007

  7. [15]

    Gormley, and Thomas Schaaf

    John Glover, Federico Fancellu, Vasudevan Jagannathan, Matthew R. Gormley, and Thomas Schaaf. 2022. https://doi.org/10.18653/v1/2022.gem-1.7 Revisiting text decomposition methods for NLI -based factuality scoring of summaries . In Proceedings of the 2nd Workshop on Natural Lan...

    2022 doi

  8. [16]

    Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. https://doi.org/10.18653/v1/N18-2017 Annotation artifacts in natural language inference data . In Proceedings of the 2018 Conference of the North A merican Chapter of the As...

    2018 doi

  9. [17]

    Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. https://doi.org/10.18653/v1/D15-1076 Question-answer driven semantic role labeling: Using natural language to annotate natural language . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processin...

    2015 doi

  10. [18]

    Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. https://doi.org/10.18653/v1/2020.acl-main.768 Are natural language inference models IMPPRESsive ? L earning IMPlicature and PRESupposition . In Proceedings of the 58th Annual Meeting of the Association f...

    2020 doi

  11. [19]

    Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.470 W i CE : Real-world entailment for claims in W ikipedia . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7561...

    2023 doi

  12. [20]

    Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations

    2019

  13. [21]

    Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81--93

    1938

  14. [22]

    Ayal Klein, Eran Hirsch, Ron Eliav, Valentina Pyatkin, Avi Caciularu, and Ido Dagan. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.528 QAS em parsing: Text-to-text modeling of QA -based semantics . In Proceedings of the 2022 Conference on Empirical Methods in Natural Langu...

    2022 doi

  15. [23]

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428

    2024 arXiv

  16. [24]

    Smith, and Yejin Choi

    Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.508 WANLI : Worker and AI collaboration for natural language inference dataset creation . In Findings of the Association for Computational Linguistics: EMNLP 202...

    2022 doi

  17. [25]

    Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439

    2023 arXiv

  18. [26]

    Bill MacCartney. 2009. Natural language inference. Stanford University

    2009

  19. [27]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generatio...

    2023 doi

  20. [28]

    Ani Nenkova and Rebecca Passonneau. 2004. https://aclanthology.org/N04-1019 Evaluating content selection in summarization: The pyramid method . In Proceedings of the Human Language Technology Conference of the North A merican Chapter of the Association for Computational Lingui...

    2004

  21. [29]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. https://doi.org/10.18653/v1/2020.acl-main.441 Adversarial NLI : A new benchmark for natural language understanding . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    2020 doi

  22. [30]

    OpenAI . 2024. https://openai.com/index/hello-gpt-4o/ Hello gpt-4o: A new model for openai's future . Accessed: 2024-10-13

    2024

  23. [31]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  24. [32]

    Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alexia Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, and Samuel R. Bowman. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.421 Does putting a linguist in the loop improve NLU data collection...

    2021 doi

  25. [33]

    Terence Parsons. 1990. Events in the semantics of english: A study in subatomic semantics

    1990

  26. [34]

    Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.713 Unsupervised question decomposition for question answering . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing...

    2020 doi

  27. [35]

    Valentina Pyatkin, Paul Roit, Julian Michael, Yoav Goldberg, Reut Tsarfaty, and Ido Dagan. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.108 Asking it all: Generating contextualized questions for any semantic role . In Proceedings of the 2021 Conference on Empirical Method...

    2021 doi

  28. [36]

    Raymond Reiter. 1980. A logic for default reasoning. Artificial intelligence, 13(1-2):81--132

    1980

  29. [37]

    Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A

    Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.418 Thinking like a skeptic: Defeasible inference in natural language . In Findings of the As...

    2020 doi

  30. [38]

    Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI conference on artificial intelligence, v...

    2019

  31. [39]

    Neha Srikanth, Marine Carpuat, and Rachel Rudinger. 2024 a . How often are errors in natural language reasoning due to paraphrastic variability? Transactions of the Association for Computational Linguistics, 12:1143--1162

    2024

  32. [40]

    Neha Srikanth and Rachel Rudinger. 2022. https://doi.org/10.18653/v1/2022.naacl-main.350 Partial-input baselines show that NLI models can ignore context, but they don ' t. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational ...

    2022 doi

  33. [41]

    Neha Srikanth, Rupak Sarkar, Heran Mane, Elizabeth Aparicio, Quynh Nguyen, Rachel Rudinger, and Jordan Boyd-Graber. 2024 b . https://doi.org/10.18653/v1/2024.naacl-long.403 Pregnant questions: The importance of pragmatic awareness in maternal health question answering . In Pro...

    2024 doi

  34. [42]

    Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Oana-Maria Camburu, and Marek Rei. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.569 Atomic inference for NLI with generated facts as atoms . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pr...

    2024 doi

  35. [43]

    Joe Stacey, Pasquale Minervini, Haim Dubossarsky, and Marek Rei. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.251 Logical reasoning with span-level predictions for interpretable and robust NLI models . In Proceedings of the 2022 Conference on Empirical Methods in Natural ...

    2022 doi

  36. [44]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  37. [45]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. https://doi.org/10.18653/v1/N18-1074 FEVER : a large-scale dataset for fact extraction and VER ification . In Proceedings of the 2018 Conference of the North A merican Chapter of the Associatio...

    2018 doi

  38. [46]

    Etsuji Tomita, Akira Tanaka, and Haruhisa Takahashi. 2006. The worst-case time complexity for generating all maximal cliques and computational experiments. Theoretical computer science, 363(1):28--42

    2006

  39. [47]

    Dhruv Verma, Yash Kumar Lal, Shreyashee Sinha, Benjamin Van Durme, and Adam Poliak. 2023. https://doi.org/10.18653/v1/2023.acl-short.76 Evaluating paraphrastic robustness in textual entailment models . In Proceedings of the 61st Annual Meeting of the Association for Computatio...

    2023 doi

  40. [48]

    Miriam Wanner, Seth Ebner, Zhengping Jiang, Mark Dredze, and Benjamin Van Durme. 2024. A closer look at claim decomposition. arXiv preprint arXiv:2403.11903

    2024 arXiv

  41. [49]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

  42. [50]

    Yating Wu, Ritika Mangla, Greg Durrett, and Junyi Jessy Li. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.325 QUD eval: The evaluation of questions under discussion discourse parsing . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processin...

    2023 doi

  43. [51]

    Moy Yuan and Andreas Vlachos. 2024. https://doi.org/10.18653/v1/2024.kallm-1.11 Zero-shot fact-checking with semantic triples and knowledge graphs . In Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024), pages 105--115, Bangkok, Thailand...

    2024 doi

  44. [52]

    Xiutian Zhao, Ke Wang, and Wei Peng. 2024. Measuring the inconsistency of large language models in preferential ranking. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 171--176

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.