REVIEW 5 cited by
Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Benchmark datasets play a central role in the organization of machine learning research. They coordinate researchers around shared research problems and serve as a measure of progress towards shared goals. Despite the foundational role of benchmarking practices in this field, relatively little attention has been paid to the dynamics of benchmark dataset use and reuse, within or across machine learning subcommunities. In this paper, we dig into these dynamics. We study how dataset usage patterns differ across machine learning subcommunities and across time from 2015-2020. We find increasing concentration on fewer and fewer datasets within task communities, significant adoption of datasets from other tasks, and concentration across the field on datasets that have been introduced by researchers situated within a small number of elite institutions. Our results have implications for scientific evaluation, AI ethics, and equity/access within the field.
Forward citations
Cited by 5 Pith papers
-
LLM Performance for Code Generation on Noisy Tasks
LLMs solve heavily obfuscated benchmark tasks, and performance decay under obfuscation differs sharply between old and new datasets, which the authors interpret as a signature of training-data contamination.
-
Completeness of Datasets Documentation on ML/AI repositories: an Empirical Investigation
Most popular ML/AI datasets are poorly documented, especially regarding collection, processing, and maintenance, according to a manual audit of 100 datasets across four repositories.
-
Making Sense of Data in the Wild: Data Analysis Automation at Scale
A multi-agent LLM system with retrieval-augmented generation automatically curates datasets from Zenodo and Hugging Face, yielding small retrieval gains and a confounded synthetic-data improvement.
-
Rethinking Indic AI from a Lens of Cultural Heritage Preservation
The paper surveys Indic NLP evolution and proposes 'Culture Sensing' to integrate indigenous oral knowledge into foundation models for cultural preservation.
-
Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
A meta-review of about 110 critical studies finds nine systemic weaknesses in AI benchmarking and concludes that benchmarks are receiving disproportionate trust in AI governance.
Discussion (0). Continue with ORCID to comment.