Pith. sign in

REVIEW 2 major objections 6 minor 2 cited by

Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models

T0 review · 2 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read NLI benchmarks still separate strong from weak LLMs, and model–human agreement on ambiguous cases improves as models scale.

desk verdict The JSD scale effect is the paper's real news, but the underspecified model distribution behind it makes that finding the one to verify before betting on it. read the letter →

arxiv 2411.14103 v1 pith:F6F3YNDU submitted 2024-11-21 cs.CL

Lovish Madaan , David Esiobu , Pontus Stenetorp , Barbara Plank , Dieuwke Hupkes This is my paper · ORCID
classification cs.CL
keywords naturallanguageinferenceLLMevaluationbenchmarksaturationJensen-ShannondivergenceChaosNLIhumanlabeldistributionsmodelscaletrainingdynamics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Natural language inference benchmarks were once the standard way to test whether a model understands language, but they have largely vanished from LLM evaluation suites. This paper asks whether that abandonment was premature, and answers yes: across five NLI benchmarks and six models from the Llama and Mistral families, accuracy separates models of different size and quality, develops over pre-training, and is not inflated by training-data contamination. The paper also takes a distributional view, using ChaosNLI's 100-way human labels to measure the Jensen-Shannon divergence between a model's softmax distribution and the human label distribution. It finds that this divergence decreases with scale and during training, yet even the largest models remain much further from human labels than two human populations are from each other. That gap is the paper's main new signal: NLI evaluation still has headroom, and model–human divergence is a statistic worth tracking.

What carries the argument

The central object is the Jensen-Shannon divergence (JSD) between a model's softmax distribution over NLI labels and the human label distribution recorded in ChaosNLI, a benchmark supplying 100 human annotations per example for subsets of MNLI, SNLI, and αNLI. JSD is a symmetric, bounded (0 to 1) distance between two probability distributions, chosen because, unlike KL divergence, it treats the two distributions symmetrically. The paper also relies on a choice-based evaluation protocol (computing negative log-likelihood over letter options) and on the monotonicity statistic (Kendall tau against an idealized increasing curve) to judge whether a benchmark can track training progress.

What would settle it

If a larger model evaluated on ChaosNLI subsets showed a JSD no lower than a smaller model's, or if JSD rose during later training while accuracy kept improving, the claim that JSD is a scale-sensitive evaluation signal would be falsified. A complementary check would be to pre-train an 8B and a 70B model on a standard public data mixture beyond 2 trillion tokens to see whether NLI accuracy still rises late in training.

Watch

Extended reading notes

Core claim

NLI tasks still provide a discriminating, non-saturated, and contamination-free signal for evaluating LLMs. Concretely, on ANLI, even the largest models remain below 70% accuracy, and across MNLI, SNLI, and HANS, accuracy gaps between 8B, 70B, and 405B models are substantial, while αNLI saturates early. A second discovery is distributional: when a model's softmax probabilities over entailment, neutral, and contradiction are compared with the full human label distribution (100 annotations per example from ChaosNLI), the Jensen-Shannon divergence is far lower than chance or earlier BERT-scale models, and — contrary to earlier findings — it decreases steadily both with model scale and over the course of pre-training. The paper interprets this as evidence that NLI benchmarks remain informative for model development and that the JSD between model and human labels is a promising, scale-sensitive evaluation statistic.

Load-bearing premise

The training-dynamics and JSD-vs-scale conclusions rest on a single from-scratch pre-training run for each of two model sizes, stopped at 2 trillion tokens using a custom, unreleased data mixture; if that run is not representative of typical pre-training, the curves in Figures 2 and 5 may not generalize.

Editorial extensions

If this is right

  • NLI benchmarks still separate models of different size and quality: on ANLI the best models stay below 70% accuracy, while MNLI and SNLI show clear gaps across the Llama and Mistral families.
  • Performance on these benchmarks develops during pre-training, so they can track long-horizon training progress, with the caveat that their monotonicity at 8B scale is low.
  • Contamination does not inflate the scores: 8-gram overlap analysis shows virtually no estimated performance gain from contaminated examples.
  • On ambiguous examples, model disagreement with the human majority label is largest when human entropy is high, and models align more with the majority label as they scale.
  • The JSD between model softmax distributions and human label distributions decreases with both scale and training time, yet stays far above human–human JSD, so there is headroom for improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit: if JSD to human labels keeps falling with scale, it could become a training-time monitoring signal or even a target, nudging models toward calibrated uncertainty rather than majority-label accuracy.
  • The authors note the jury idea in passing; a consequence they do not spell out is that a panel of LLM judges may share systematic biases, so jury–human agreement should be checked against human label distributions before using panels for evaluation.
  • Because the training dynamics rest on one custom datamix, a natural test is to re-run the 8B and 70B curves on a standard public mix; the paper's monotonicity table suggests NLI accuracy may not be reliable for checkpoint selection at small scales regardless.
  • A testable extension: if JSD is a genuine quality signal, models fine-tuned or post-trained to minimize JSD on ChaosNLI should also become better at selective prediction and calibration under distribution shift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper asks whether NLI benchmarks remain informative for evaluating large language models. It evaluates five NLI benchmarks (SNLI, MNLI, HANS, ANLI, αNLI) on six base models (Llama-3.1 8B/70B/405B and Mistral 7B, Mixtral 8x7B/8x22B) using a multiple-choice NLL scoring setup. The reported results are that accuracy discriminates model scale, few-shot examples are needed for reasonable performance, accuracy develops during pre-training (with low monotonicity for the 8B model), contamination does not meaningfully inflate scores, and errors on the strongest model often coincide with human disagreement. Using ChaosNLI, the paper compares model probability distributions with human label distributions via Jensen-Shannon divergence (JSD), reporting that JSD decreases with scale and during training, although model-human JSD remains substantially above human-human JSD. The central recommendation is that NLI benchmarks, and especially distributional agreement with human labels, remain useful signals for LLM evaluation.

Significance. If the JSD finding is robust, it provides a potentially practical training-monitoring statistic and a way to quantify alignment with human label variation, directly extending the earlier ChaosNLI result that model-human JSD did not improve with scale. The discriminability, few-shot, and contamination analyses are useful and would help practitioners decide whether to include NLI benchmarks in evaluation suites. The paper makes good use of public benchmarks and an established contamination methodology, and the manual error analysis connects accuracy saturation to genuine label ambiguity rather than model deficiency. However, the headline JSD claim currently depends on an unspecified and unvalidated proxy for model label distributions, and the training-dynamics claims rest on single from-scratch runs with a custom unreleased datamix. These issues need to be resolved before the central conclusions can be taken at face value.

major comments (2)
  1. [§4.5, with §3.3 and Table 5] The stress-test concern about the JSD computation lands. Section 4.5 reports Jensen-Shannon divergences between model softmax distributions and human label distributions, but the paper never states how the model distribution over labels is obtained. The evaluation in §3.3 computes only the negative log-likelihood of the option letter ('A'/'B'/'C'), so the natural reading is that JSD is computed from a softmax over these letter NLLs. This proxy is not validated: it ignores probability mass of the full answer string, is sensitive to tokenization of single letters, and is vulnerable to position and letter priors. Because the prompt template in Table 5 always orders the options as A. Entailment, B. Neutral, C. Contradiction, any systematic letter preference can inflate or deflate JSD in a way that correlates with scale. The paper reports no checks with permuted option orders or with full-label scoring. Please specify the exact distribution used and add such checks; without them, the headline scale and training effects in Figures 4b and 5 could be artifacts of the proxy rather than evidence about human-aligned distributions.
  2. [§4.2, Figure 5, Appendix D] The training-dynamics and JSD-during-training claims rest on a single from-scratch pre-training run each for the 8B and 70B models, stopped at 2T tokens with the authors explicitly noting that they did not have the budget for a full run. Appendix D describes the data only as a custom mix of public web, code, and reasoning data, without sufficient detail for replication. One run per scale provides no estimate of run-to-run variance, and the unreleased datamix makes it impossible for others to assess how representative the curves are. Please either add multiple runs or seeds, use checkpoints from a released model series that span training, or clearly reframe these sections as a single-configuration case study and soften the general conclusion that JSD decreases steadily during training.
minor comments (6)
  1. [Conclusion] The sentence listing the benchmarks says 'five different NLI benchmarks – αNLI, ANLI, HANS, and MNLI', omitting SNLI; the list should include all five.
  2. [§4.5] The text refers to 'the three possible labels Entailment, Neutral, and Contradiction', but αNLI is a two-choice task and the ChaosNLI αNLI subset presumably has two labels; clarify how the JSD is computed for the binary tasks.
  3. [Figure 1 and footnote 3] The x-axis labeled 'shots' should be made explicit as 'number of few-shot examples', and the number of examples used for each shot setting should be stated in the caption or main text rather than only in a footnote.
  4. [§4.4] The manual error analysis covers 40 predictions from a single model (Llama-3.1 405B) on MNLI only; the generalization that 'most of the mistakes in MNLI' are human-disagreement cases should be hedged accordingly.
  5. [§4.3] The phrase 'contamination does not play a participatory role' is unclear; use 'does not play a significant role' or similar, and fix the typo 'indicitive'.
  6. [§2.1] The sentence 'the benchmarks MNLI and RTE and RTE were used' appears to have a duplicated phrase; remove the repetition.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline results are direct measurements on public benchmarks, and the few self-citations supply measurement tools rather than load-bearing conclusions.

full rationale

The headline results are direct measurements on fixed public benchmarks. Accuracy, NLL, and JSD are computed by scoring models against SNLI, MNLI, HANS, ANLI, alphaNLI, and ChaosNLI labels; no parameter is fitted to the target conclusions, and no quantity is defined in terms of the outcome it is used to establish. The self-citations in Section 4.2 (Madaan et al. 2024 for the Kendall-tau monotonicity definition) and Section 4.3 (Singh et al. 2024 for the ConTAM contamination methodology) supply measurement definitions and tools rather than load-bearing empirical premises; the contamination analysis is performed on independently trained from-scratch models, and the monotonicity statistic is a standard rank correlation. The JSD comparison in Section 4.5 uses model softmax scores and human label distributions from ChaosNLI; the softmax-over-letters operationalization may be a validity concern (for example, letter-position bias could confound scale effects), but that is a measurement-robustness issue, not a circular reduction. The from-scratch training runs are incomplete and use a custom datamix, but that is a data limitation rather than an instance of the target result being assumed. I therefore find no circular step.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities, forces, or conserved quantities. Its claims are empirical measurements over existing benchmarks, and the central findings depend on standard evaluation assumptions rather than invented constructs. The main ledger items are domain assumptions about what NLI accuracy and softmax distributions measure.

assumptions (5)
  • domain assumption Entailment and contradiction are central to language meaning, making NLI a valid test of language understanding.
    Invoked in the Introduction, Section 1, citing Bowman et al. (2015); underpins the relevance of the whole study.
  • domain assumption Choice-based NLL over answer letters is a valid proxy for the model's label distribution.
    Section 3.3 defines predictions via NLL over letters; Section 4.5 treats the resulting softmax as the model's distribution for JSD comparisons with human labels.
  • domain assumption The 8-gram overlap contamination score and ConTAM thresholds provide a valid contamination estimate.
    Section 4.3 follows Singh et al. (2024) and Dubey et al. (2024); the validity of the method is assumed rather than re-derived.
  • domain assumption The human-human JSD reported by Nie et al. (2020b) is a valid baseline for human disagreement.
    Section 4.5 compares model-human JSD to the human-human JSD computed in prior work on the same data.
  • domain assumption A single from-scratch 2T-token pre-training run per scale represents typical training dynamics.
    Section 4.2 and Appendix D; the runs are single, partial, and use an unreleased custom datamix, so representativeness is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models." pith.science (2026). https://pith.science/paper/F6F3YNDU

@misc{pith2026241114103,
  author       = {Pith},
  title        = {Pith review of: Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F6F3YNDU}},
  note         = {Machine review of arXiv:2411.14103}
}
read the original abstract

In the recent past, a popular way of evaluating natural language understanding (NLU), was to consider a model's ability to perform natural language inference (NLI) tasks. In this paper, we investigate if NLI tasks, that are rarely used for LLM evaluation, can still be informative for evaluating LLMs. Focusing on five different NLI benchmarks across six models of different scales, we investigate if they are able to discriminate models of different size and quality and how their accuracies develop during training. Furthermore, we investigate the extent to which the softmax distributions of models align with human distributions in cases where statements are ambiguous or vague. Overall, our results paint a positive picture for the NLI tasks: we find that they are able to discriminate well between models at various stages of training, yet are not (all) saturated. Furthermore, we find that while the similarity of model distributions with human label distributions increases with scale, it is still much higher than the similarity between two populations of humans, making it a potentially interesting statistic to consider.

Figures

Figures reproduced from arXiv: 2411.14103 by the authors.

Figure 1
Figure 1. Performance across shots. We show the accuracies for six fully pre-trained models on the five NLI benchmarks. Dashed lines indicate random and finetuned-BERT baselines. (a) ANLI (b) HANS (c) MNLI (d) SNLI (e) αNLI [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Performance during pre-training. We show how accuracy for the five benchmarks develops during pre-training for two Llama-3 style models. the benchmark scores during training – on both a discrete (accuracy) and continuous metric (NLL). In [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Contamination results. We show the EPG vs the percent of the evaluation dataset marked as contaminated according to different thresholds. observed at those threshold is likely an artefact of the small size of the clean partition, and this result is likely not indicative of true performance gain. Thus, we believe that contamination does not play a participatory role in high performances for NLI benchmarks. 4.4 Datase… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Accuracy vs entropy and final model JSDs a) Accuracy vs entropy for Llama8B and Llama 405B. We show how the accuracy of Llama 8B and Llama 405B changes as the entropy of the human label distributions increases. Accuracy-vs-entropy plots for all other models can be foun…
Figure 5
Figure 5. Figure 5: Development of JSD during training. We show how the JSD of our trained-from-scratch 8B and 70B model develops during training. et al., 1997) to measure the distance between the two distributions. Contrary to KL divergence, JSD is symmetric and bound between 0 and 1, ma…
Figure 6
Figure 6. Figure 6: Accuracy versus entropy. We show how the accuracy of Llama-3.1 70B and the Mistral series of models changes as the entropy of the human label distributions increases. B Entropy vs accuracy plots In addition to Figure 4a highlighting the results on the Llama-3.1 8B and …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Math Natural Language Inference: this should be easy!

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A new Math NLI corpus from category theory abstracts plus a ten-model evaluation shows LLM unanimous votes approach human labels (88%) but individual LLMs still make basic math reasoning errors.

  2. A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI

    cs.CL 2024-12 conditional novelty 6.0 of 10

    LLM-generated explanations, paired with a few human labels, produce model judgment distributions as close to human judgment distributions as human explanations do on NLI.

Reference graph

Works this paper leans on

37 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Lora Aroyo and Chris Welty. 2015. https://api.semanticscholar.org/CorpusID:6134326 Truth is a lie: Crowd truth and the seven myths of human annotation . AI Mag., 36:15--24

    work page 2015

  4. [4]

    Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.124 Stop measuring calibration when humans disagree . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892--1915, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

    doi:10.18653/v1/2022.emnlp-main.124 2022

  5. [5]

    Joris Baan, Raquel Fern \'a ndez, Barbara Plank, and Wilker Aziz. 2024. https://aclanthology.org/2024.eacl-short.24 Interpreting predictive probabilities: Model confidence or human label variation? In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 268--277, St. Ju...

    work page 2024

  6. [6]

    Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. 2020. https://openreview.net/forum?id=Byg1v1HKDB Abductive commonsense reasoning . In International Conference on Learning Representations

    work page 2020

  7. [7]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. https://doi.org/10.18653/v1/D15-1075 A large annotated corpus for learning natural language inference . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632--642, Lisbon, Portugal. Association for Computational Linguistics

    doi:10.18653/v1/d15-1075 2015

  8. [8]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

Show all 37 references
  1. [9]

    Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. 2024. https://arxiv.org/abs/2406.17600 ``seeing the big through the small'': Can llms approximate human judgment distributions on nli from a few explanations? Preprint, arXiv:2406.17600

    2024 arXiv

  2. [10]

    Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pages 177--190. Springer

    2005

  3. [11]

    Manning, and Christopher Potts

    Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2012. https://doi.org/10.1162/COLI_a_00097 Did it happen? the pragmatic complexity of veridicality assessment . Computational Linguistics, 38(2):301--333

    2012 doi

  4. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    2019 arXiv

  5. [13]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  6. [14]

    Ritam Dutt, Sagnik Ray Choudhury, Varun Venkat Rao, Carolyn Rose, and V.G.Vinod Vydiswaran. 2024. https://aclanthology.org/2024.genbench-1.11 Investigating the generalizability of pretrained language models across multiple dimensions: A case study of NLI and MRC . In Proceedin...

    2024

  7. [15]

    Clémentine Fourrier, Nathan Habib, Julien Launay, and Thomas Wolf. 2023. https://huggingface.co/blog/open-llm-leaderboard-mmlu What's going on with the open llm leaderboard?

    2023

  8. [16]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  9. [17]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  10. [18]

    Noah Lee, Na Min An, and James Thorne. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.278 Can large language models capture dissenting human voices? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4569--4585, Singapore. Assoc...

    2023 doi

  11. [19]

    Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes

    Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. 2024. https://arxiv.org/abs/2406.10229 Quantifying variance in evaluation benchmarks . Preprint, arXiv:2406.10229

    2024 arXiv

  12. [20]

    Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. https://doi.org/10.18653/v1/P19-1334 Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3...

    2019 doi

  13. [21]

    Mar \' a Luisa Men \'e ndez, JA Pardo, L Pardo, and MC Pardo. 1997. The jensen-shannon divergence. Journal of the Franklin Institute, 334(2):307--318

    1997

  14. [22]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020 a . https://doi.org/10.18653/v1/2020.acl-main.441 Adversarial NLI : A new benchmark for natural language understanding . In Proceedings of the 58th Annual Meeting of the Association for C...

    2020 doi

  15. [23]

    Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020 b . https://doi.org/10.18653/v1/2020.emnlp-main.734 What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMN...

    2020 doi

  16. [24]

    Xenia Ohmer, Elia Bruni, and Dieuwke Hupkes. 2023. https://aclanthology.org/2023.gem-1.22 Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses . In Proceedings of the Third Workshop on Natural Language Generation, Evaluation...

    2023

  17. [25]

    Xenia Ohmer, Elia Bruni, and Dieuwke Hupkes. 2024. https://doi.org/10.1162/coli_a_00529 From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency . Computational Linguistics, pages 1--51

    2024 doi

  18. [26]

    Ellie Pavlick and Tom Kwiatkowski. 2019. https://doi.org/10.1162/tacl_a_00293 Inherent disagreements in human textual inferences . Transactions of the Association for Computational Linguistics, 7:677--694

    2019 doi

  19. [27]

    Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.731 The `` problem '' of human label variation: On ground truth in data, modeling and evaluation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671--10682, Ab...

    2022 doi

  20. [28]

    Barbara Plank, Dirk Hovy, and Anders S gaard. 2014. https://doi.org/10.3115/v1/E14-1078 Learning part-of-speech taggers with inter-annotator agreement loss . In Proceedings of the 14th Conference of the E uropean Chapter of the Association for Computational Linguistics , pages...

    2014 doi

  21. [29]

    Adam Poliak. 2020. https://doi.org/10.18653/v1/2020.eval4nlp-1.10 A survey on recognizing textual entailment as an NLP evaluation . In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 92--109, Online. Association for Computational Linguistics

    2020 doi

  22. [30]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. https://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer . Preprint, arXiv:1910.10683

    2023 arXiv

  23. [31]

    Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes

    Aaditya K. Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes. 2024. https://arxiv.org/abs/2411.03923 Evaluation data contamination in llms: how do we measure it and (when) does it matter? Preprint, arXiv:2411.03923

    2024 arXiv

  24. [32]

    Pat Verga, Sebastian Hofst \" a tter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick S. H. Lewis. 2024. https://doi.org/10.48550/ARXIV.2404.18796 Replacing judges with juries: Evaluating LLM generations with a panel o...

    2024 doi arXiv

  25. [33]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019 a . https://proceedings.neurips.cc/paper_files/paper/2019/file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf Superglue: A stickier benchmark for general-p...

    2019

  26. [34]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019 b . https://openreview.net/forum?id=rJ4km2R5t7 GLUE: A multi-task benchmark and analysis platform for natural language understanding . In 7th International Conference on Learning Repr...

    2019

  27. [35]

    Lucas Weber, Elia Bruni, and Dieuwke Hupkes. 2023. https://doi.org/10.18653/v1/2023.conll-1.20 Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning . In Proceedings of the 27th Conference on Computational Natural Language Learni...

    2023 doi

  28. [36]

    Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. https://aclanthology.org/2024.acl-long.123 V ari E rr NLI : Separating annotation error from human label variation . In Proceedings of the 62nd Annual Meeting of the Association for Computatio...

    2024

  29. [37]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.