REVIEW 2 cited by
Benchmarking a Benchmark: How Reliable is MS-COCO?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Benchmark datasets are used to profile and compare algorithms across a variety of tasks, ranging from image classification to segmentation, and also play a large role in image pretraining algorithms. Emphasis is placed on results with little regard to the actual content within the dataset. It is important to question what kind of information is being learned from these datasets and what are the nuances and biases within them. In the following work, Sama-COCO, a re-annotation of MS-COCO, is used to discover potential biases by leveraging a shape analysis pipeline. A model is trained and evaluated on both datasets to examine the impact of different annotation conditions. Results demonstrate that annotation styles are important and that annotation pipelines should closely consider the task of interest. The dataset is made publicly available at https://www.sama.com/sama-coco-dataset/ .
Forward citations
Cited by 2 Pith papers
-
Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluation
PoVisLE, a manually constructed Polish cultural visual question-answering benchmark, shows current vision-language models reach at most 71.45% accuracy and perform worst on dialect and regionalism questions.
-
Pseudo-Labeling Driven Refinement of Benchmark Object Detection Datasets via Analysis of Learning Patterns
MJ-COCO, a pseudo-labeling based re-annotation of MS-COCO, improves detection on some external benchmarks but reduces performance on the standard MS-COCO validation set.
Discussion (0). Continue with ORCID to comment.