Pith. sign in

REVIEW 4 cited by

Adversarial NLI: A New Benchmark for Natural Language Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1910.14599 v2 pith:DDYNGZM5 submitted 2019-10-31 cs.CL cs.LG

Yixin Nie , Adina Williams , Emily Dinan , Mohit Bansal , Jason Weston , Douwe Kiela This is my paper
classification cs.CLcs.LG
keywords benchmarkadversarialdatasetmodelsstate-of-the-artanalysisannotatorsapplied
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure. We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks, while posing a more difficult challenge with its new test set. Our analysis sheds light on the shortcomings of current state-of-the-art models, and shows that non-expert annotators are successful at finding their weaknesses. The data collection method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability

    cs.CL 2025-05 conditional novelty 7.0 of 10

    Rationale-augmented finetuning can hurt accuracy while improving calibration, with the sizes of both effects tied linearly to task difficulty.

  2. Towards Compute-Optimal Many-Shot In-Context Learning

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Hybrid demonstration selection that adds 20 similar examples to a large cached random or k-means set matches or beats similarity-only selection at up to 10x lower estimated inference cost in many-shot ICL.

  3. DELIVER: A System for LLM-Guided Coordinated Multi-Robot Pickup and Delivery using Voronoi-Based Relay Planning

    cs.RO 2025-08 conditional novelty 4.0 of 10

    DELIVER combines an LLM parser, Voronoi territory division, and relay handoffs so multiple robots can cooperatively deliver an item from a single spoken instruction.

  4. Agentic Web: Weaving the Next Web with AI Agents

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A position paper defines the Agentic Web as the next web era and proposes a three-dimensional conceptual framework for understanding and building it.

Pith tools