Pith. sign in

REVIEW 4 cited by

Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.09269 v3 pith:JXIT3OYH submitted 2024-09-14 cs.CV cs.AIcs.CLcs.LG

Neelabh Sinha , Vinija Jain , Aman Chadha This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords frameworkmodelstypesvlmsapplicationdomainsevaluationknowledge
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Visual Question-Answering (VQA) has become key to user experience, particularly after improved generalization capabilities of Vision-Language Models (VLMs). But evaluating VLMs for an application requirement using a standardized framework in practical settings is still challenging. This paper aims to solve that using an end-to-end framework. We present VQA360 - a novel dataset derived from established VQA benchmarks, annotated with task types, application domains, and knowledge types, for a comprehensive evaluation. We also introduce GoEval, a multimodal evaluation metric developed using GPT-4o, achieving a correlation factor of 56.71% with human judgments. Our experiments with state-of-the-art VLMs reveal that no single model excels universally, thus, making a right choice a key design decision. Proprietary models such as Gemini-1.5-Pro and GPT-4o-mini generally outperform others, but open-source models like InternVL-2-8B and CogVLM-2-Llama-3-19B also demonstrate competitive strengths, while providing additional advantages. Our framework can also be extended to other tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Separating Clicks from Baits: Using Large Language Models to Detect Misleading YouTube Thumbnails

    cs.SI 2026-07 conditional novelty 6.0 of 10

    Claude 3.5 Sonnet with dynamic few-shot multi-modal prompts detects misleading YouTube thumbnails at 93.8% accuracy on a balanced cross-country dataset, outperforming CHECKER and open-weight VLMs.

  2. ThumbnailTruth: A Multi-Modal LLM Approach for Detecting Misleading YouTube Thumbnails Across Diverse Cultural Settings

    cs.SI 2025-09 conditional novelty 6.0 of 10

    Claude 3.5 Sonnet, prompted with thumbnails, subtitles, and video summaries, detects misleading YouTube thumbnails with up to 93.8% accuracy on a new cross-country dataset.

  3. mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A systematic empirical study finds that for multimodal RAG, EVA-CLIP retrieval, listwise LVLM reranking, and feeding only the top-ranked document works best, with a self-reflection agent adding further gains.

  4. Exploring Primitive Visual Measurement Understanding and the Role of Output Format in Learning in Vision-Language Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Fine-tuning vision-language models with sentence-formatted outputs instead of tuple outputs improves shape attribute and coordinate prediction for larger models, and scaling the loss on numeric tokens sharpens numeric...

Pith tools