REVIEW 1 cited by
SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
This paper presents an augmentation of MSCOCO dataset where speech is added to image and text. Speech captions are generated using text-to-speech (TTS) synthesis resulting in 616,767 spoken captions (more than 600h) paired with images. Disfluencies and speed perturbation are added to the signal in order to sound more natural. Each speech signal (WAV) is paired with a JSON file containing exact timecode for each word/syllable/phoneme in the spoken caption. Such a corpus could be used for Language and Vision (LaVi) tasks including speech input or output instead of text. Investigating multimodal learning schemes for unsupervised speech pattern discovery is also possible with this corpus, as demonstrated by a preliminary study conducted on a subset of the corpus (10h, 10k spoken captions). The dataset is available on Zenodo: https://zenodo.org/record/4282267
Forward citations
Cited by 1 Pith paper
-
MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models
On 190 tasks and a 12-direction cross-modal diagnostic, seven embedding models frequently fail to honor explicit target-modality instructions: retrieval is biased toward the query modality and instruction-induced shif...
Discussion (0). Continue with ORCID to comment.