Pith. sign in

REVIEW 10 cited by

Beyond Accuracy: Behavioral Testing of NLP models with CheckList

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2005.04118 v1 pith:AMJ37KAY submitted 2020-05-08 cs.CL cs.LG

Marco Tulio Ribeiro , Tongshuang Wu , Carlos Guestrin , Sameer Singh This is my paper
classification cs.CLcs.LG
keywords checklistmodelstesttestingaccuracybehavioralbugscommercial
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors. Inspired by principles of behavioral testing in software engineering, we introduce CheckList, a task-agnostic methodology for testing NLP models. CheckList includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation, as well as a software tool to generate a large and diverse number of test cases quickly. We illustrate the utility of CheckList with tests for three tasks, identifying critical failures in both commercial and state-of-art models. In a user study, a team responsible for a commercial sentiment analysis model found new and actionable bugs in an extensively tested model. In another user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy

    cs.AI 2026-05 conditional novelty 6.0 of 10

    Models flip between correct and incorrect answers on over 23% of questions under meaning-preserving paraphrases, so single-prompt accuracy overstates reliable knowledge.

  2. Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation

    cs.RO 2025-08 conditional novelty 6.0 of 10

    Low within-subdataset diversity and large between-subdataset differences cause shortcut learning in generalist robot policies, and targeted augmentation can mitigate it.

  3. ASSURE: Metamorphic Testing for AI-powered Browser Extensions

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A modular metamorphic testing framework for LLM-based browser extensions reports 531 automatically detected issues across six real-world extensions.

  4. Potemkin Understanding in Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LLMs frequently pass definition questions yet fail to use the same concepts in classification, generation, and editing tasks, a gap the authors call potemkin understanding.

  5. Testing Retrieval-Augmented Generation Systems with Chunk Coverage

    cs.SE 2026-07 conditional novelty 5.0 of 10

    Chunk Coverage, a suite-level, oracle-independent measure of how much of a RAG corpus a test suite retrieves, speeds up coverage growth and earlier fault discovery in clinical and financial RAG systems.

  6. Position: Stop Reactively Patching Your Model Every Time and Start Proactive Test-Driven AI Development

    cs.LG 2026-07 conditional novelty 5.0 of 10

    In a stylized model, a proactive flywheel that fixes whole groups of related scenarios needs Θ(K log K) update rounds versus Θ(M log M) for reactive patching.

  7. SPARSE Data, Rich Results: Few-Shot Semi-Supervised Learning via Class-Conditioned Image Translation

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    A GAN framework that translates unlabeled medical images between classes and fuses ensemble, time-averaged pseudo-labels outperforms six prior GAN semi-supervised methods on MedMNIST at 5-50 labels per class.

  8. GenFair: Systematic Test Generation for Fairness Fault Detection in Large Language Models

    cs.SE 2025-06 reject novelty 5.0 of 10

    GenFair generates diverse source test cases via equivalence partitioning, mutation, and boundary value analysis, then applies metamorphic relations and tone comparison to flag fairness faults, claiming higher fault de...

  9. Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    LLMs infer drug use from alcohol or smoking mentions in clinical notes, producing gender-skewed false positives that prompting only partially corrects.

  10. Agentic Web: Weaving the Next Web with AI Agents

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A position paper defines the Agentic Web as the next web era and proposes a three-dimensional conceptual framework for understanding and building it.

Pith tools