Pith. sign in

REVIEW 19 cited by

LLaVA-Critic: Learning to Evaluate Multimodal Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.02712 v2 pith:MAJRU4SR submitted 2024-10-03 cs.CV cs.CL

Tianyi Xiong , Xiyao Wang , Dong Guo , Qinghao Ye , Haoqi Fan , Quanquan Gu , Heng Huang , Chunyuan Li This is my paper
classification cs.CVcs.CL
keywords evaluationllava-criticlearningmodelmultimodalalignmentlmmsmodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce LLaVA-Critic, the first open-source large multimodal model (LMM) designed as a generalist evaluator to assess performance across a wide range of multimodal tasks. LLaVA-Critic is trained using a high-quality critic instruction-following dataset that incorporates diverse evaluation criteria and scenarios. Our experiments demonstrate the model's effectiveness in two key areas: (1) LMM-as-a-Judge, where LLaVA-Critic provides reliable evaluation scores, performing on par with or surpassing GPT models on multiple evaluation benchmarks; and (2) Preference Learning, where it generates reward signals for preference learning, enhancing model alignment capabilities. This work underscores the potential of open-source LMMs in self-critique and evaluation, setting the stage for future research into scalable, superhuman alignment feedback mechanisms for LMMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention

    cs.CV 2026-07 conditional novelty 7.0 of 10

    Gazette is the first generative gaze-to-text model: it decodes a single gaze scanpath into free-form natural-language descriptions of the viewer's goal, using GPT-4-generated 'think-aloud' transcripts as auxiliary tra...

  2. MentalThink: Shaping Thoughts in Mental SVG World

    cs.AI 2026-07 conditional novelty 7.0 of 10

    MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.

  3. AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation

    cs.CL 2025-07 conditional novelty 7.0 of 10

    With prompt engineering (audio concatenation plus in-context examples), large audio models rank speech synthesis systems in line with human preferences, reaching up to 0.91 Spearman correlation.

  4. Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark

    cs.CV 2025-06 conditional novelty 7.0 of 10

    MMRB is the first benchmark combining multi-image inputs with chain-of-thought reasoning annotations, and its evaluation shows open-source MLLMs trail commercial models while multi-image reward models are unstable.

  5. Understanding Generative AI Capabilities in Everyday Image Editing Tasks

    cs.CV 2025-05 conditional novelty 7.0 of 10

    On real Reddit photo-editing requests, human judges prefer human edits over AI edits 66% of the time, and AI editors can satisfactorily handle about 33% of requests.

  6. Poplar: A Scalable Pipeline for Human-Centric Image Dataset Synthesis

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Poplar-9K is a curated human-centric image dataset generated through a reproducible attribute sampling, diffusion rendering, and vision-language inspection pipeline, with 9,401 accepted pairs from 11,765 candidates.

  7. Improving Large Vision and Language Models by Learning from a Panel of Peers

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A panel of LVLMs that generate, evaluate, and learn from each other's outputs improves average benchmark scores by 9 points across 15 tasks.

  8. LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model

    cs.CV 2025-08 conditional novelty 6.0 of 10

    RL training on preference-labeled critic data transforms a 7B vision-language model into both a stronger critic and a stronger generative policy, improving average benchmark accuracy by 5.7% and enabling self-critique...

  9. VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answering

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A person-anchored tree plus multi-agent LLM pipeline lets a system answer cross-video queries about the same person, and it beats single-video models on the authors' new CrossVideoQA benchmark.

  10. VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A vision-expert-filtered, chain-of-thought-guided, iteratively fine-tuned reward model boosts a compact 7B model's ability to judge vision-language responses, especially detecting hallucinations.

  11. ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A verifiable RL proxy task that asks VLMs to locate a single injected hallucination in a 200-word caption improves visual perception and transfers to math and abstract reasoning benchmarks.

  12. AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A stochastic scheduling framework that modulates slow-to-fast reasoning in large reasoning models at test time, improving accuracy while reducing token usage.

  13. UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    UniGen shows a 1.5B model trained on open data can beat larger systems on image understanding and generation once it verifies its own outputs with chain-of-thought and Best-of-N selection.

  14. MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A human-annotated multimodal preference dataset plus critique-based reward modeling and reward-margin-weighted DPO improves MLLM performance across many benchmarks.

  15. LLM-as-a-Verifier: A General-Purpose Verification Framework

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Expecting over scoring-token logits yields continuous, scalable verification that improves agent trajectory selection and dense RL rewards across coding, robotics, and medical benchmarks.

  16. Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning

    cs.CV 2025-06 reject novelty 5.0 of 10

    A two-stage, value-guided decoding strategy with a margin-based reward adjustment is claimed to yield more faithful, detailed VLM captions at about a quarter of VisVM's inference cost.

  17. Generative RLHF-V: Learning Principles from Multi-modal Human Preference

    cs.AI 2025-05 conditional novelty 5.0 of 10

    A reinforcement-learned multimodal judge with grouped pairwise scoring improves vision-language model alignment on seven benchmarks.

  18. MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models

    cs.CV 2025-07 conditional novelty 4.0 of 10

    MCA-LLaVA reindexes image tokens by sums of mirrored 2D coordinates so instruction tokens attend across the whole image, reducing hallucination on POPE, CHAIR, and MME.

  19. LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs

    cs.CV 2025-06 conditional novelty 4.0 of 10

    LeanPO improves Video-LLM alignment by using a reference-free average-likelihood reward, self-generated winning/losing pairs, and dynamic label smoothing, yielding gains on six video benchmarks.

Pith tools