REVIEW 6 cited by
Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recently, there have been breakthroughs in computer vision ("CV") models that are more generalizable with the advent of models such as CLIP and ALIGN. In this paper, we analyze CLIP and highlight some of the challenges such models pose. CLIP reduces the need for task specific training data, potentially opening up many niche tasks to automation. CLIP also allows its users to flexibly specify image classification classes in natural language, which we find can shift how biases manifest. Additionally, through some preliminary probes we find that CLIP can inherit biases found in prior computer vision systems. Given the wide and unpredictable domain of uses for such models, this raises questions regarding what sufficiently safe behaviour for such systems may look like. These results add evidence to the growing body of work calling for a change in the notion of a 'better' model--to move beyond simply looking at higher accuracy at task-oriented capability evaluations, and towards a broader 'better' that takes into account deployment-critical features such as different use contexts, and people who interact with the model when thinking about model deployment.
Forward citations
Cited by 6 Pith papers
-
Scaling Vision-Language Models Is Not Enough to Mitigate Bias
Model scale loses predictive power for multi-attribute bias robustness in VLMs, while training-data size and curation remain the more reliable levers.
-
Between Gradient and Natural Gradient: A Continuum of LoRA Initializations
Gradient-projection, Adam-like, and K-FAC-whitened LoRA initializations are all special cases of one two-parameter family, and the best exponents are task-dependent and usually interior.
-
Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models
Richer text prompts from LLM synonyms and cleaner image regions from activation maps improve zero-shot vision-language classification.
-
PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projection
PRISM debiases CLIP by using an LLM to generate biased scene descriptions and then learning a linear projection of the embedding space that reduces spurious correlations, yielding higher worst-group accuracy on Waterb...
-
Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations
ReTA improves CLIP's test-time adaptation by reweighting sample selection with prediction consistency and adapting class decision boundaries via Gaussian text embeddings.
-
Learning from Limited and Imperfect Data
A doctoral thesis compiling nine peer-reviewed papers on long-tailed image generation, long-tailed recognition, semi-supervised learning, and domain adaptation.
Discussion (0). Sign in to comment.