Pith. sign in

REVIEW 4 major objections 4 minor 62 references

EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EACO claims that 5,000 self-generated images, graded by a separate Critic model, are enough to cut hallucinations by 65.6% on HallusionBench and improve reasoning by 21.8% on MME-Cognition.

desk verdict A useful but sloppy recipe for critic-guided MLLM alignment; the headline claim is misreported and the key ablation is missing. read the letter →

arxiv 2412.04903 v2 pith:GDDTOX6M submitted 2024-12-06 cs.CV cs.AIcs.CLcs.LG

Yongxin Wang , Meng Cao , Haokun Lin , Mingfei Han , Liang Ma , Jin Jiang , Yuhao Cheng , Xiaodan Liang This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords multimodallargelanguagemodelshallucinationmitigationDirectPreferenceOptimizationcriticmodelself-generateddatavisualreasoningalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EACO claims that a multimodal large language model can align itself using preference data it generates from just 5,000 unlabeled images, provided a separately trained Critic model grades the responses. The paper builds the Critic by fine-tuning an MLLM on over 137,000 scoring instructions refined from 51,000 images labeled by GPT-4V, then uses the Critic's scores to pick the best and worst of several self-generated answers. Direct Preference Optimization on those pairs, followed by an extra supervised fine-tuning stage, is reported to reduce hallucinations by 65.6% on HallusionBench and improve reasoning by 21.8% on MME-Cognition. The practical claim is that expensive human or proprietary preference labels can be replaced by a cheap open critic without sacrificing alignment quality.

What carries the argument

The load-bearing mechanism is the Critic model plus the selection rule it feeds. The Critic is an MLLM fine-tuned with LoRA on over 137,000 scoring instructions built from 51,000 images, producing scores across dimensions such as relevance, basic elements, and clarity. For each image-question pair, the target model self-generates $n$ responses; selection follows $y_w = \arg\max_j S_i^{y_j}$ and $y_l = \arg\min_j S_i^{y_j}$, where $S_i^{y_j}$ is the critic score. These pairs drive a modified DPO objective that adds a preferred-response reinforcement term and a verbosity penalty, followed by an enhanced SFT stage on DPO-improved captions. The Critic does the work of replacing expensive human or proprietary preference labels with a model trained on GPT-4V-labeled data.

What would settle it

Run the full EACO pipeline on the same 5,000 images but replace the Critic's scores with random rankings of the same self-generated responses; if HallusionBench and MME-Cognition still improve by the reported margins, the critic-guided selection is not the cause of the gains.

Watch

Extended reading notes

Core claim

The central claim is that a critic-guided self-preference loop is enough to meaningfully align an MLLM. Starting from a target model, the method generates multiple responses per image-question pair, has a Critic model trained on refined GPT-4V-scored data assign quality scores, and selects the highest-scoring response as preferred and the lowest as non-preferred. These pairs are used in a modified DPO objective with two extra regularizers, one reinforcing the preferred response and one penalizing verbosity, and the resulting model is then given an enhanced SFT pass on its own DPO-improved captions. On LLaVA-v1.6-Mistral-7B this yields a 65.6% relative reduction on HallusionBench and a 21.8% gain on MME-Cognition, with consistent improvements also reported for Bunny-8B and MiniCPM-V. The paper's claim is therefore that open-source critic models can stand in for human and proprietary labelers, making preference alignment economical and transferable across architectures.

Load-bearing premise

The load-bearing assumption is that the separate evaluation model called the Critic, trained on GPT-4V-labeled data, ranks the target model's own self-generated responses reliably enough that the highest- and lowest-scored outputs really are the better and worse answers; if the scores are noisy or biased toward wording rather than visual facts, the preference pairs become near-random and the reported gains would come from the extra SFT stage instead.

Editorial extensions

If this is right

  • Preference data for MLLM alignment no longer needs to be bought from human annotators or proprietary APIs: 5,000 unlabeled images and a fine-tuned critic are enough to reproduce large benchmark gains.
  • Hallucination control and reasoning improvement can come from the same tuning recipe, since the reported gains appear simultaneously on HallusionBench and MME-Cognition.
  • Because the same procedure improves LLaVA-v1.6, Bunny-8B, and MiniCPM-V, the alignment effect transfers across model families rather than being an artifact of one backbone.
  • The ablation's diminishing returns beyond 5,000 samples and after the first iteration imply that a single critic-guided preference pass captures most of the available benefit for this data budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not ablate the two extra DPO regularizers separately, so a natural next test is to run EACO with only the verbosity penalty or only the preferred-response reinforcement; this would show how much of the gain is due to critic-ranked pairs rather than the modified objective.
  • The reported self-rewarding baseline is much weaker than the Critic version, which suggests the Critic's training data, not the self-generation loop, is what creates the improvement; swapping the Critic for GPT-4V scores on the same 5,000 images would quantify how close the open critic is to proprietary labeling.
  • The 65.6% HallusionBench reduction may be tied to the visual-faithfulness dimension in the Silkie-derived scoring rubric; a testable extension is to check whether the Critic's dimension scores individually predict gains on different hallucination categories, such as existence versus attribute errors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes EACO, a framework that trains a critic model on GPT-4V-annotated instruction data, uses that critic to score self-generated responses from a base MLLM, constructs preference pairs from the highest- and lowest-scoring responses, and then performs DPO with a modified objective (Equation 2) followed by an enhanced supervised fine-tuning stage. The method is evaluated on LLaVA-v1.6-Mistral-7B, Bunny-8B, and MiniCPM-V-8B, reporting consistent improvements across comprehensive, domain-specific, and hallucination benchmarks, with the largest gains on hallucination benchmarks.

Significance. If fully validated, EACO would offer an economical (5k images) and model-agnostic alignment recipe that reduces reliance on proprietary reward models while improving reasoning and hallucination metrics. The paper provides source code, and the reported gains are consistent across three different base architectures, which strengthens the empirical claim. The principal limitation is that the unique contribution of critic-guided DPO is not isolated from the post-DPO SFT stage, so the current evidence does not pin down the mechanism responsible for the gains.

major comments (4)
  1. [Abstract and Section 5.2] The abstract states that EACO 'reduces the overall hallucinations by 65.6% on HallusionBench.' This is inconsistent with Table 1, where the HallusionBench score increases from 29.1 (LLaVA-v1.6-7B) to 48.2 (EACO). A relative increase of 65.6% in the accuracy metric is not a 65.6% reduction in hallucinations; the corresponding reduction in hallucination rate would be (70.9 - 51.8)/70.9 ≈ 26.9%. The current wording misstates the direction and magnitude of the effect and should be corrected in the abstract and in Section 5.2 where the same claim appears.
  2. [Section 5.3 (ablation study)] The paper attributes the reported improvements to critic-guided DPO, but the full pipeline also includes an enhanced SFT stage after DPO (Section 4.3). The ablation study in Section 5.3 varies data scale, iteration count, critic prompt style, and self-rewarding critic, but never removes the DPO component or the enhanced SFT component. Without conditions such as enhanced-SFT-only, DPO-only, random preference pairs, or reversed critic scores, the specific contribution of the critic-guided DPO to the observed gains is not established. This attribution gap is load-bearing because the central claim is about critic-guided preference alignment.
  3. [Section 4.3, Equation (2)] The modified DPO objective in Equation (2) includes the term -(α|yw| - α|yl|) inside the log-sigmoid, which is described as discouraging verbose responses. As written, this term penalizes only the length difference between the preferred and non-preferred responses; it does not penalize verbosity of the generated output in an absolute sense. If the intended regularization is to discourage long responses, the term should depend on absolute lengths (e.g., -α(|yw|+|yl|) or a length penalty appended outside the log-sigmoid). Please clarify the formulation and, if the equation is a typographical error, correct it, because the stated behavior of the objective does not match the equation.
  4. [Section 5.3, Self-Rewarding Model paragraph] The text states that 'the self-rewarding method shows some minor improvements in certain metrics, such as MME P improving from 1512.3 to 1506.2.' However, Table 1 reports MME P of 1506.2 for Self-Rewarding, which is lower than the baseline 1512.3, not an improvement. This is a factual contradiction that should be fixed.
minor comments (4)
  1. [Throughout] There are numerous typos and inconsistent abbreviations; examples include 'Multilingual Language Model' in Section 3.2 (should be 'Multimodal'), 'reasonging' in Figure 1, 'Perferred' in Figure 4, 'Imapct' in Section 5.3, 'Scaling up dateset' in Section 5.3, and 'Ohters' in Table 1's footnote. A careful proofreading pass is recommended.
  2. [Algorithm 1 and Equation (2)] The notation for the reference model is inconsistent: the algorithm line shows 'pref (yw|x,v)' where a subscript or a clear reference-model notation (e.g., p_ref) is needed. Please align the notation between Algorithm 1 and Equation (2).
  3. [Section 5.3 and Figure 5] The text reports a gain of 'about 8.5%' for 5k samples, but Figure 5's left panel shows gains in the range 7–9% for data scales; please make sure the axis labels and numbers are consistent, and specify whether the percentages are relative or absolute.
  4. [Supplementary Section 7] The qualitative comparison of critics (Tables 3–6) shows that the LLaVA-v1.6 model, used as a critic, assigns perfect scores to flawed responses. This is an interesting observation, but the main text does not report any quantitative reliability measure of the trained critic (e.g., agreement with GPT-4V on a held-out set). Adding such a measure would support the assumption that the critic's rankings are meaningful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the critic is trained on external GPT-4V-labeled Silkie data, preference pairs are chosen by critic scores (Eq. 1), and gains are measured on external benchmarks; no claimed result reduces by construction to its inputs.

full rationale

I walked the derivation chain: the Critic is LoRA-fine-tuned on roughly 137k scoring instructions refined from Silkie's GPT-4V annotations (Sec. 3.1-3.2); EACO self-generates n responses, scores them with this Critic, and sets yw = argmax and yl = argmin (Eq. 1); DPO (Eq. 2) optimizes the policy against these pairs, and the subsequent enhanced SFT uses the model's own post-DPO captions following STIC. No step uses the evaluation benchmarks (MME, SEED, ScienceQA, MathVista, POPE, AMBER, HallusionBench) in the training or selection objective, and the paper explicitly states it checks that preference/SFT images do not overlap critic training images. The only flagged items are the acknowledged similarity to LLaVA-Critic and the lack of an ablation isolating critic-guided DPO from enhanced SFT; these are attribution and novelty concerns, not circularity. The critic is trained on external labels rather than on the target model's outputs or on the reported metrics; no equation makes the predicted benchmark improvement identical to a training signal, no fitted parameter is renamed as a prediction, and no load-bearing claim rests on a self-citation. A single incidental self-citation ([4], cited only as an example MLLM) is not load-bearing.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on training-data quality, critic transfer, the correctness of the DPO objective, self-training validity, and a benchmark aggregation choice. No new physical or mathematical entities are introduced.

free parameters (4)
  • DPO regularization alpha = 1e-3
    Appears in Equation (2) and in the supplementary hyperparameter table; set by the authors to balance the preferred-response and verbosity-penalty terms.
  • DPO beta (KL temperature) = not reported
    Standard DPO coefficient required to reproduce the loss; missing from the supplementary hyperparameter table, so exact reproduction is not possible.
  • Critic score-gap filter threshold = not reported
    Section 3.1 keeps response pairs 'with a larger score gap' but gives no numeric cutoff, changing the composition of the 137k critic training set.
  • Self-generated response count n = not reported
    Section 4.1 samples n responses per prompt and Equation (1) selects max/min over n, but the value of n is never specified.
assumptions (5)
  • domain assumption Silkie's GPT-4V ratings are valid supervision for visual response quality.
    The critic training set in Section 3.1 is a refined version of Silkie, whose labels were produced by GPT-4V; teacher bias propagates to the critic.
  • domain assumption A critic trained on score-gap-filtered pairs generalizes to the target model's self-generated responses.
    Section 4.2 assumes the critic can rank responses from LLaVA-v1.6, Bunny-8B, and MiniCPM-V even though its training responses come from the 12 models in Silkie; no transfer analysis is reported.
  • ad hoc to paper The modified DPO objective in Equation (2) implements the described regularization.
    The printed negative sign on the preferred-response term contradicts the prose in Section 4.3, and no code is provided to resolve which version was used.
  • domain assumption Enhanced SFT on the model's own post-DPO captions is a beneficial training step.
    Section 4.3 follows STIC and assumes the DPO-enhanced descriptions are high-quality targets, even though they come from the model being trained.
  • ad hoc to paper The average of seven normalized benchmarks is a valid single performance measure.
    Section 5.3 rescales each benchmark maximum to 100 and averages seven benchmarks to compute 61.106 and 66.327, but does not list the seven benchmarks or the normalization details.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation." pith.science (2026). https://pith.science/paper/GDDTOX6M

@misc{pith2026241204903,
  author       = {Pith},
  title        = {Pith review of: EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDDTOX6M}},
  note         = {Machine review of arXiv:2412.04903}
}
read the original abstract

Multimodal large language models (MLLMs) have achieved remarkable progress on various visual question answering and reasoning tasks leveraging instruction fine-tuning specific datasets. They can also learn from preference data annotated by human to enhance their reasoning ability and mitigate hallucinations. Most of preference data is generated from the model itself. However, existing methods require high-quality critical labels, which are costly and rely on human or proprietary models like GPT-4V. In this work, we propose Enhancing Alignment in MLLMs via Critical Observation (EACO), which aligns MLLMs by self-generated preference data using only 5k images economically. Our approach begins with collecting and refining a Scoring Evaluation Instruction-tuning dataset to train a critical evaluation model, termed the Critic. This Critic observes model responses across multiple dimensions, selecting preferred and non-preferred outputs for refined Direct Preference Optimization (DPO) tuning. To further enhance model performance, we employ an additional supervised fine-tuning stage after preference tuning. EACO reduces the overall hallucinations by 65.6% on HallusionBench and improves the reasoning ability by 21.8% on MME-Cognition. EACO achieves an 8.5% improvement over LLaVA-v1.6-Mistral-7B across multiple benchmarks. Remarkably, EACO also shows the potential critical ability in open-source MLLMs, demonstrating that EACO is a viable path to boost the competence of MLLMs.

Figures

Figures reproduced from arXiv: 2412.04903 by the authors.

Figure 1
Figure 1. Upper: Response examples from original LLaVA-v1.6 7B[25] and LLaVA-v1.6 7B w/ EACO, which mitigates hallucination and improve reasonging ability. Lower Left: The framework of EACO. The process begins with an image-question pair, which is fed to the ini￾tialized MLLM to genrate multiple responses. And then these responses are evaluated by a Critic Model that provides judgments regarding their quality. Based on the cr… view at source ↗
Figure 2
Figure 2. Various datasets used for scoring evaluation. Each dataset contains a specific number of instructions, with a total of 51,000 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Critic Model training pipeline. After filtering of critic [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of preferred and non-preferred responses generated for two visual content summarization examples. The [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Ablation Studies on Preference Dataset Scaling, Critic [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 14 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 2

    arXiv 2023

  3. [3]

    Introducing our multimodal models, 2023

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa ˘gnak Tas ¸ırlar. Introducing our multimodal models, 2023. 2

    2023

  4. [4]

    Continual llava: Continual instruction tuning in large vision-language models

    Meng Cao, Yuyang Liu, Yingfei Liu, Tiancai Wang, Ji- ahua Dong, Henghui Ding, Xiangyu Zhang, Ian Reid, and Xiaodan Liang. Continual llava: Continual instruction tuning in large vision-language models. arXiv preprint arXiv:2411.02564, 2024. 2

    arXiv 2024

  5. [5]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 2

    arXiv 2023

  6. [6]

    Pca-bench: Evaluating multimodal large language models in perception-cognition-action chain.arXiv preprint arXiv:2402.15527, 2024

    Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, and Baobao Chang. Pca-bench: Evaluating multimodal large language models in perception-cognition-action chain.arXiv preprint arXiv:2402.15527, 2024. 3

    arXiv 2024

  7. [7]

    Lawrence Zit- nick

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zit- nick. Microsoft coco captions: Data collection and evalu- ation server. arXiv preprint arXiv:1504.00325, 2015. 2, 6

    arXiv 2015

  8. [8]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023. 2, 4

    arXiv 2023

Show all 62 references
  1. [9]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023. 2

    2023 arXiv

  2. [10]

    Enhancing large vision language models with self-training on image comprehension

    Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self-training on image comprehension. arXiv preprint arXiv:2405.19716, 2024. 1, 3, 5, 6, 7

    2024 arXiv

  3. [11]

    What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning

    Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jin- peng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, and Ji-Rong Wen. What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning. arXiv preprint arXiv:2311.01487, 2023. 3

    2023 arXiv

  4. [12]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings 9 of the 32nd ACM International Conference on Multimed...

    2024

  5. [13]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 2, 7, 1

    2023 arXiv

  6. [14]

    Hallusionbench: An advanced diag- nostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: An advanced diag- nostic suite for entangled language hallucination and visual illusion in large vision-language models. arXiv ...

    arXiv

  7. [15]

    Efficient multi- modal learning from data-centric perspective

    Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multi- modal learning from data-centric perspective. arXiv preprint arXiv:2402.11530, 2024. 6, 7

    2024 arXiv

  8. [16]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 4, 6

    2021 arXiv

  9. [17]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 2, 7, 1

    2023 arXiv

  10. [18]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890, 2023. 2, 3

    2023 arXiv

  11. [19]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 2

    2023 arXiv

  12. [20]

    Silkie: Preference distillation for large visual lan- guage models

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual lan- guage models. arXiv preprint arXiv:2312.10665, 2023. 1, 2, 3, 4, 7

    2023 arXiv

  13. [21]

    M3it: A large-scale dataset towards multi- modal multilingual instruction tuning

    Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M3it: A large-scale dataset towards multi- modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387, 2023. 2, 3

    2023 arXiv

  14. [22]

    Red teaming visual language models

    Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, and Qi Liu. Red teaming visual language models. arXiv preprint arXiv:2401.12915, 2024. 3

    2024 arXiv

  15. [23]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 7, 2

    2023 arXiv

  16. [24]

    Aligning large multi-modal model with robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023. 2, 3, 4

    2023 arXiv

  17. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 1, 2, 3, 4, 6, 7, 5, 8

    2023 arXiv

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023. 2

    2023

  19. [27]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024. 2, 4

    2024 arXiv

  20. [28]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  21. [29]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts. InIn- ternational Conference on Learning Represen...

  22. [30]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024. 3

    2024 arXiv

  23. [31]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 4

    2023 arXiv

  24. [32]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Ad- vances in neural information processing systems, 35:...

    2022

  25. [33]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  26. [34]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. 2, 3, 6

    2024

  27. [35]

    Proximal policy optimization algo- rithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms. arXiv preprint arXiv:1707.06347, 2017. 3

    2017 arXiv

  28. [36]

    Aligning large multi- modal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi- modal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023. 1, 2, 3, 4, 7

    2023 arXiv

  29. [37]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2

    2023 arXiv

  30. [38]

    Eyes wide shut? exploring the 10 visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the 10 visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 4

    2024

  31. [39]

    An llm-free multi-dimensional benchmark for mllms hallu- cination evaluation

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free multi-dimensional benchmark for mllms hallu- cination evaluation. arXiv preprint arXiv:2311.07397, 2023. 7, 2

    2023 arXiv

  32. [40]

    En- hancing visual-language modality alignment in large vi- sion language models via self-improvement

    Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, and Cao Xiao. En- hancing visual-language modality alignment in large vi- sion language models via self-improvement. arXiv preprint arXiv:2...

    2024 arXiv

  33. [41]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 2

    2024 arXiv

  34. [42]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 35:24824–24837, 2022. 9

    2022

  35. [43]

    Llava- critic: Learning to evaluate multimodal models

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava- critic: Learning to evaluate multimodal models. arXiv preprint arXiv:2410.02712, 2024. 2, 3

    2024 arXiv

  36. [44]

    Vigor: Improving visual ground- ing of large vision language models with fine-grained reward modeling

    Siming Yan, Min Bai, Weifeng Chen, Xiong Zhou, Qixing Huang, and Li Erran Li. Vigor: Improving visual ground- ing of large vision language models with fine-grained reward modeling. arXiv preprint arXiv:2402.06118, 2024. 4

    2024

  37. [45]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 6, 7

    2024 arXiv

  38. [46]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional hu- man feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional hu- man feedback. In Proceedings of the IEEE/CVF Conference on ...

    2024

  39. [47]

    Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220, 2024. 3, 4, 7, 1

    2024

  40. [48]

    Self-rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason We- ston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024. 7, 1, 6

    2024 arXiv

  41. [49]

    Pmc-vqa: Vi- sual instruction tuning for medical visual question answer- ing

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Vi- sual instruction tuning for medical visual question answer- ing. arXiv preprint arXiv:2305.10415, 2023. 3

    2023 arXiv

  42. [50]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023. 3, 4

    2023 arXiv

  43. [51]

    Svit: Scaling up visual instruction tuning

    Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04087, 2023. 3, 4

    2023 arXiv

  44. [52]

    Aligning modalities in vision large lan- guage models via preference fine-tuning

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large lan- guage models via preference fine-tuning. arXiv preprint arXiv:2402.11411, 2024. 3, 7, 1

    2024 arXiv

  45. [53]

    Calibrated self-rewarding vision language models

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622 , 2024. 3, 7, 1

    2024 arXiv

  46. [54]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2 11 EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation Supp...

    2023 arXiv

  47. [55]

    GPT-4o and our Critic model produce similar scores for responses, but they fail to identify the flaws in bad responses from the baseline LLA V A model

    The Comparison of Critics Here, we present some examples of the comparison of crit- ics from different models in Table 3, 4, 5, and 6. GPT-4o and our Critic model produce similar scores for responses, but they fail to identify the flaws in bad responses from the baseline LLA V A model

  48. [56]

    As shown in Table 8, most of the experiment is con- ducted with prompts in rating style, apart from the ablation study presented in Section 5.3

    Critic Prompts In the experiment, we utilize prompts in three different styles. As shown in Table 8, most of the experiment is con- ducted with prompts in rating style, apart from the ablation study presented in Section 5.3

  49. [57]

    The training details are shown in Table 2

    Training Details The training comprises the Critic model training, DPO training, and enhanced SFT. The training details are shown in Table 2

  50. [58]

    Using annotated preference data, one round of preference learning is conducted on LLaV A1.5

    Baseline Details LLaV A-RLHF[36] proposes a novel alignment algorithm called Factually Augmented RLHF that enhances the re- ward model by incorporating additional factual data, such as image captions and ground-truth multiple-choice options. Using annotated preference data, on...

  51. [59]

    Benchmark Details MME [13] serves as a comprehensive benchmark, aiming to assess perceptual and cognitive capability of MLLMs within 14 sub-tasks. SEEDBench [17] is a comprehensive benchmark that evaluates the hierarchical capabilities of MLLMs, including the generation of bot...

  52. [60]

    Critic Dataset Here, we will present some examples of Critic Dataset for Critic Model training in Table 2

  53. [61]

    Here, we will show some examples between EACO and baseline LLaV A-v1.6- Mistral-7B in Table 9 and 10

    The Comparison between Models In the experiments, EACO achieves remarkable improve- ment on hallucination reduction. Here, we will show some examples between EACO and baseline LLaV A-v1.6- Mistral-7B in Table 9 and 10. 2 Example 2: The comparison of critic from multiple models...

  54. [62]

    score:⟨total points⟩

    High Quality: The response is well-tailored to the image, demonstrating expert-level insight and attention to detail without including irrelevant information. Score: 5. score: 25 Table 5. The analysis of three different models—GPT-4o, a proprietary Critic Model, and LLaV A-v1....

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.