REVIEW 4 cited by
CLAIR: Evaluating Image Captions with Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The evaluation of machine-generated image captions poses an interesting yet persistent challenge. Effective evaluation measures must consider numerous dimensions of similarity, including semantic relevance, visual structure, object interactions, caption diversity, and specificity. Existing highly-engineered measures attempt to capture specific aspects, but fall short in providing a holistic score that aligns closely with human judgments. Here, we propose CLAIR, a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) to evaluate candidate captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Notably, on Flickr8K-Expert, CLAIR achieves relative correlation improvements over SPICE of 39.6% and over image-augmented methods such as RefCLIP-S of 18.3%. Moreover, CLAIR provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score. Code is available at https://davidmchan.github.io/clair/
Forward citations
Cited by 4 Pith papers
-
SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation
A new reference-free metric, SPECS, fine-tunes LongCLIP with a specificity objective and reaches LLM-level human correlation on long captions at a fraction of the computational cost.
-
CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioning
A single LLaVA-based captioning model continuously controls caption length, descriptiveness, and word uniqueness by interpolating between learned endpoint conditioning vectors.
-
TriPSS: A Tri-Modal Keyframe Extraction Framework Using Perceptual, Structural, and Semantic Representations
TriPSS fuses CIELAB color, ResNet-50, and LLaMA caption embeddings through PCA and HDBSCAN to extract keyframes, reporting F1 of 0.6104 on TVSum20 and 0.5902 on SumMe.
-
IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth
A visual heatmap tool lets people rate vision-language model reliability in video by inspecting patterns of green and red cells, with user ratings tracking objective F1 scores when those exist.
Discussion (0). Continue with ORCID to comment.