Pith. sign in

REVIEW 3 cited by

Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.02287 v1 pith:IYP6PC34 submitted 2024-05-03 cs.CL cs.AIcs.CV

Piotr Padlewski , Max Bain , Matthew Henderson , Zhongkai Zhu , Nishant Relan , Hai Pham , Donovan Ong , Kaloyan Aleksiev
show 14 more authors
Aitor Ormazabal Samuel Phua Ethan Yeo Eugenie Lamprecht Qi Liu Yuqi Wang Eric Chen Deyu Fu Lei Li Che Zheng Cyprien de Masson d'Autume Dani Yogatama Mikel Artetxe Yi Tay
This is my paper
classification cs.CLcs.AIcs.CV
keywords modelsevaluationvibe-evalautomatichardhumanmultimodalchallenging
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce Vibe-Eval: a new open benchmark and framework for evaluating multimodal chat models. Vibe-Eval consists of 269 visual understanding prompts, including 100 of hard difficulty, complete with gold-standard responses authored by experts. Vibe-Eval is open-ended and challenging with dual objectives: (i) vibe checking multimodal chat models for day-to-day tasks and (ii) rigorously testing and probing the capabilities of present frontier models. Notably, our hard set contains >50% questions that all frontier models answer incorrectly. We explore the nuances of designing, evaluating, and ranking models on ultra challenging prompts. We also discuss trade-offs between human and automatic evaluation, and show that automatic model evaluation using Reka Core roughly correlates to human judgment. We offer free API access for the purpose of lightweight evaluation and plan to conduct formal human evaluations for public models that perform well on the Vibe-Eval's automatic scores. We release the evaluation code and data, see https://github.com/reka-ai/reka-vibe-eval

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Current vision-language models are largely miscalibrated when they verbalize confidence, visual reasoning models such as o3 and o4-mini are better calibrated, and Visual Confidence-Aware Prompting reduces ECE on IsoBench.

  2. ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

    cs.CV 2025-02 conditional novelty 6.0 of 10

    ZeroBench is a hand-built 100-question visual reasoning benchmark, adversarially filtered so every evaluated frontier LMM scored 0% at release.

  3. MiMo-VL Technical Report

    cs.CL 2025-06 conditional novelty 5.0 of 10

    MiMo-VL-7B-RL, a 7B open-source vision-language model, reports state-of-the-art results on 35 of 40 benchmarks and a 59.4 OlympiadBench score, with the report crediting long-CoT pretraining data and mixed on-policy RL.

Pith tools