Pith. sign in

REVIEW 1 cited by

Data Contamination Report from the 2024 CONDA Shared Task

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.21530 v2 pith:HR2UAN54 submitted 2024-07-31 cs.CL cs.LG

Oscar Sainz , Iker García-Ferrero , Alon Jacovi , Jon Ander Campos , Yanai Elazar , Eneko Agirre , Yoav Goldberg , Wei-Lin Chen
show 20 more authors
Jenny Chim Leshem Choshen Luca D'Amico-Wong Melissa Dell Run-Ze Fan Shahriar Golchin Yucheng Li Pengfei Liu Bhavish Pahwa Ameya Prabhu Suryansh Sharma Emily Silcock Kateryna Solonko David Stap Mihai Surdeanu Yu-Min Tseng Vishaal Udandarao Zengzhi Wang Ruijie Xu Jinglin Yang
This is my paper · ORCID
classification cs.CLcs.LG
keywords contaminationdatasharedtaskcommunityevaluationassistavailable
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform. The platform continues to be online, open to contributions from the community.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation

    cs.CL 2025-01 conditional novelty 7.0 of 10

    Contaminating pre-training data with full source-target translation test pairs inflates BLEU scores, much more for 8B models than 1B models, while partial contamination has smaller and less consistent effects.

Pith tools