Pith. sign in

REVIEW 4 major objections 5 minor 72 references

LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LeanPO replaces DPO's ratio-based reward with the policy model's own average response likelihood, stopping the likelihood-displacement collapse in Video-LLM alignment and lifting LLaVA-NeXT-Video-7B from 38.8 to 43.6 on Video-MME.

desk verdict Useful data pipeline and plausible gains, but the equations as written make the loss a constant; major revision needed. read the letter →

arxiv 2506.05260 v1 pith:P5I3RYXP submitted 2025-06-05 cs.CV

Xiaodong Wang , Jinfa Huang , Li Yuan , Peixi Peng This is my paper
classification cs.CV
keywords preferenceoptimizationvideolargelanguagemodelslikelihooddisplacementdirectself-generateddatadynamiclabelsmoothingquestionansweringhallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the main obstacle to efficient preference alignment of Video-LLMs is likelihood displacement: during DPO training the probabilities of both the chosen and the rejected response fall, because the objective only rewards the gap between them. Its proposal, LeanPO, replaces DPO's ratio-of-likelihoods reward with the policy model's own length-averaged log-likelihood, making the winning response's reward stable instead of falling, and it pairs this reward with a data pipeline that keeps likelihood and trustworthiness aligned: the winning response is generated with the ground-truth answer as a hint and refined by self-reflection, while the losing response comes from a video-augmented pass. A dynamic label-smoothing rule down-weights the rare pairs where the losing response accidentally outscores the winner. If the claims hold, video preference alignment becomes cheaper and more stable, with no reference model and no external reward labels, and the paper reports LLaVA-NeXT-Video-7B rising from 38.8 to 43.6 on Video-MME and from 53.9 to 59.7 on NeXT-QA, plus consistent gains on LLaVA-Video, Qwen2-VL, and Qwen2.5-VL checkpoints.

What carries the argument

The load-bearing object is the reformulated implicit reward, the length-normalized average log-likelihood $r(V,q,y)=\frac{\beta}{|y|}\sum_{i=1}^{|y|}\log\pi_\theta(y_i\mid [V,q], y_{<i})$, which replaces DPO's reference-ratio reward and makes the training objective a margin maximization over likelihoods. Around it sits the reward-trustworthiness-correlated self-generated preference data pipeline: for each video-text triplet $(V,q,a)$ the model is prompted with the ground-truth answer as a hint, produces an initial response, and then rewrites that response in a self-reflection pass to form the winning answer $y_w$, while the losing answer $y_l$ is generated from an augmented copy of the video; the design goal is that $y_w$ carries both higher likelihood and higher trustworthiness than $y_l$. The third component is dynamic label smoothing: a pseudo-label $z_q$ is set to 1 when the likelihood margin falls below a threshold $d$, indicating a possibly noisy pair, and the smoothed preference probability $\tilde{p}(y_w\succ y_l)=(1-z_q\alpha)\,p(y_w\succ y_l)+z_q\alpha\,p(y_l\succ y_w)$ replaces the raw Bradley-Terry term, with a margin term $\gamma$ further separating the two responses in Eq. 4. These three pieces, reward form, pair construction, and dynamic labels, carry the argument, and each is ablated in Table 6.

What would settle it

Take a preference pair in which the human-preferred response has a lower average log-likelihood than the rejected one (Fig. 3 shows such inversions are routine for ground-truth answers), optimize the LeanPO loss on it, and record the chosen response's likelihood: if it decreases, the average-likelihood reward is misaligned with quality outside the pipeline's specially generated pairs, and the reported gains would depend on the GT-hint data construction rather than on the reward reformulation alone.

Watch

Extended reading notes

Core claim

The paper's central claim, stated on its own terms, is that likelihood displacement is more severe for Video-LLMs than for LLMs: the redundant complexity of video inputs pushes both $\log\pi_\theta(y_w\mid x)$ and $\log\pi_\theta(y_l\mid x)$ down faster during DPO, and the root cause is the ratio form of the DPO reward, $\log(\pi_\theta(y)/\pi_{\mathrm{sft}}(y))$, which becomes erratic when both likelihoods fall and the gap to the SFT model widens. LeanPO therefore redefines the implicit reward as the average likelihood of the response under the policy model itself, $r(V,q,y) = \frac{\beta}{|y|}\sum_{i=1}^{|y|}\log\pi_\theta(y_i\mid [V,q], y_{<i})$, turning the training objective into a direct maximization of the likelihood margin between winning and losing responses. Because ground-truth answers, despite being most trustworthy, receive the lowest likelihood (Fig. 3), the paper argues the reward is reliable only for pairs in which likelihood and trustworthiness are correlated, and contributes a pipeline that manufactures such pairs. The final piece is a dynamic label-smoothing objective in which a pseudo-label $z_q$ flags pairs whose reward margin falls below a threshold $d$, and the smoothed preference probability $(1-z_q\alpha)\,p(y_w\succ y_l) + z_q\alpha\,p(y_l\succ y_w)$ replaces the raw Bradley-Terry term.

Load-bearing premise

The method rests on the premise that a response's average likelihood under the model is a dependable measure of its quality, but the paper's own Fig. 3 shows the most trustworthy answers (the ground truths) receive the lowest likelihood, so the reward only tracks trustworthiness for the specially engineered pairs the pipeline creates.

Editorial extensions

If this is right

  • The reference-free reward removes the need to keep the SFT reference model during alignment, and the paper shows the winning response's reward curve stays stable or rises during training instead of falling.
  • Any existing video QA or video caption corpus can be converted into paired preference data by the pipeline, without human annotators, GPT judges, or a reward model; the paper uses 17k triplets from LLaVA-Hound and trains on a single pass.
  • The gains transfer across model families and capacities: LLaVA-NeXT-Video-7B, LLaVA-Video-7B, Qwen2-VL-7B, Qwen2.5-VL-7B, and the image model LLaVA-v1.5-7B all improve on the reported benchmarks.
  • The largest reported gains land on the temporal-reasoning benchmarks (Video-MME and NeXT-QA), the settings where the paper argues displacement and hallucination are most visible.
  • Scaling the preference dataset further improves results, with the medium-duration Video-MME slice benefiting most as data grows (Fig. 5).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its reward is calibrated only inside its own pipeline: Fig. 3 shows ground-truth answers, despite maximal trustworthiness, receive the lowest likelihood, so running LeanPO unchanged on human- or GPT-labeled pairs would test whether the reward generalizes, and it would likely push chosen responses downhill whenever the chosen answer is a low-likelihood ground truth.
  • If the explanation that video redundancy amplifies displacement is right, the severity of likelihood displacement should scale with input complexity: more frames, longer context, or denser temporal events should produce larger likelihood drops, which the paper does not measure directly.
  • The dynamic label smoothing can be viewed as an online noise filter, which suggests a simpler alternative worth testing: discard pairs whose margin falls below the threshold $d$ instead of smoothing them, at lower implementational cost.
  • Because the winning response is always generated with the ground-truth answer as a hint, the pipeline inherits the quality of the source annotations; on subjectively judged or open-ended video questions, the reflection step could propagate annotation bias rather than remove it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LeanPO, a reference-free preference optimization method for Video-LLMs. The key idea is to replace the DPO-style ratio reward with the average log-likelihood of a response under the policy model, pair it with a self-generated preference data pipeline that injects ground-truth hints into winning responses and uses video augmentation for losing responses, and add a dynamic label-smoothing rule based on the reward margin. The authors report consistent gains over two video-LLM baselines on multiple-choice and open-ended benchmarks, including improvements of 12.4% on Video-MME and 10.8% on NeXT-QA for LLaVA-NeXT-Video-7B, plus generalization experiments on Qwen-VL and image-based LLaVA models.

Significance. If the formulation is corrected and the empirical claims hold, LeanPO is a practical and inexpensive alignment recipe: it removes the reference model, avoids external reward models or GPT-based labeling, and shows gains across several backbones and benchmarks. The authors also release code and test on both video and image LLMs, which strengthens the paper's reproducibility and scope. The main caveats are that the central objective as written is non-differentiable, that the reward reformulation is essentially SimPO, and that several hyperparameters and tuning choices are underreported, so the specific novelty and the robustness of the reported margins need to be clarified.

major comments (4)
  1. [Sec. 3.2, Eqs. (2), (3), and (7)] The reward in Eqs. (2) and (3) is defined with the frozen SFT model π_sft, but the final objective (7) optimizes π_θ. Taken literally, r(V,q,y_w) and r(V,q,y_l) do not depend on θ, so p(y_w ≻ y_l) and the pseudo-label z_q are constants with respect to θ and L_LEANPO has zero gradient. The abstract and Section 1 explicitly state that the reward is the average likelihood under the policy model, so this is very likely a typo, but as written the central training objective is unimplementable. Please correct Eqs. (2) and (3) to use π_θ and verify that all subsequent equations are consistent.
  2. [Sec. 3.2, Eq. (4) and Related Work Sec. 2.2] After replacing π_sft with π_θ in Eqs. (2)-(3), Eq. (4) is exactly SimPO's reference-free objective with a margin γ: -log σ(β/|y_w| log π_θ(y_w) - β/|y_l| log π_θ(y_l) - γ). The paper's claim that reformulating the reward as the average likelihood is a contribution is therefore not correct, since SimPO already uses this reward. The genuinely new components are the self-generated data pipeline and the dynamic label smoothing in Eqs. (5)-(6). The paper should acknowledge this overlap explicitly and position the contribution accordingly.
  3. [Sec. 3.1 and Fig. 3] Fig. 3 shows that ground-truth answers, which the paper itself regards as having the highest trustworthiness, receive the lowest average-likelihood reward. This directly contradicts the premise that the average likelihood is a reliable proxy for response quality. The reward-trustworthiness correlation is therefore not a property of the reward itself but an artifact of the constructed data pipeline, where GT hints are injected into winning responses and augmentation is used for losing responses. The authors should either provide evidence that likelihood and trustworthiness are generally aligned on unmodified model outputs, or restrict the claim to the proposed data distribution and explain why the method would transfer to other preference-data sources.
  4. [Sec. 4.1, Tab. 7, and Fig. 5] The reward margin γ and the label-flipping threshold d appear in Eqs. (4)-(5) but are never reported in the implementation details or the appendix. In addition, the smoothing factor α is selected in Tab. 7 by evaluating on Video-MME, which is the same benchmark used to report the final gains; this is test-set tuning and can inflate the results. All reported numbers appear to be single runs with no error bars or significance tests; given that some improvements are small (e.g., 0.8% overall for LLaVA-Video), the authors should provide multiple seeds or at least standard deviations, and should evaluate α on a held-out validation set rather than on Video-MME.
minor comments (5)
  1. [Sec. 4.2, Tables 3 and 9] Several table entries have missing spaces or fused numbers, e.g., '28.238.422.3' in Table 3 and '59.855.9' in Table 9; please reformat the tables for readability.
  2. [Fig. 5 caption] The caption contains a typo: 'for on Video-MME' should be 'on Video-MME'. Also, the figure reports a single trajectory with one run per size, so it is unclear whether the non-monotonic medium-duration behavior is significant.
  3. [References] References [15] and [21] are the same DPO-shift paper, and references [16] and [20] are the same 'Preference fine-tuning of LLMs should leverage suboptimal, on-policy data' paper; duplicate citations should be removed.
  4. [Sec. 4.1 and Appendix A] The text refers to 'More details are shown in Appendix A', but Appendix A only describes the benchmarks and contains no implementation details, hyperparameter settings, or training configurations; please either add the missing details or fix the cross-reference.
  5. [Sec. 2.2 and Sec. 3.2] The notation [V,q] is used inconsistently for conditioning in Eqs. (2)-(3) and for the concatenation of the video and question in Eq. (1); please define the notation once and use it uniformly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation chain; the benchmark results are external evidence, while the noted pi_sft/pi_theta inconsistency is a specification bug rather than circularity.

full rationale

The paper's derivation chain is not circular. The claimed reformulation of the reward as average likelihood is the training objective itself, which is self-referential only in the same way any preference loss is: the loss is constructed to maximize the reward margin, and the reported gains are measured on independent benchmarks rather than read off from that construction. Section 3.1's data pipeline injects the ground-truth answer into the winning response and uses augmented videos for the losing response; this is a supervised data-generation procedure, and the statement that 'For almost any triplet ... r(V,q,y_w)>r(V,q,y_l)' is an empirical property of that pipeline, not a prediction forced by the equations. The dynamic label smoothing in Eqs. (5)-(6) uses the same reward to build pseudo-labels; this is algorithmic self-referencing, not derivational circularity. There is no load-bearing self-citation and no fitted parameter renamed as a prediction. Two concerns surfaced in review are real but outside the circularity definition: Eqs. (2)-(3) write the reward with the frozen SFT model pi_sft, which would make Eq. (7) constant with respect to theta as written (a specification/implementation defect), and, if the reward is instead computed with pi_theta as the text claims, the reward coincides with the already-cited SimPO objective, reducing the novelty of that component. Neither defect makes the empirical benchmark improvements equivalent to the paper's inputs; the comparisons in Tables 2, 5, and 9 remain independent external evidence.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method relies on standard preference optimization assumptions plus two domain-specific assumptions about the data pipeline: GT-hint injection improves winning responses, and augmentation degrades losing responses. The most fragile input is the unverified claim that likelihood is a trustworthy reward, given the paper's own Fig. 3 shows GT answers with the lowest likelihood. No new physical or conceptual entities are introduced.

free parameters (5)
  • beta (reward scaling) = not reported
    Scaling constant in Eq. 2-3 for the average-likelihood reward; value not specified in the paper.
  • gamma (reward margin) = not reported
    Margin term in Eq. 4 that controls the separation between winning and losing rewards; value not reported.
  • d (label flipping threshold) = not reported
    Threshold in Eq. 5 for deciding when to flip the preference label; value not reported.
  • alpha (label smoothing factor) = 0.1 (best on Video-MME)
    Label smoothing factor in Eq. 6; Tab. 7 selects 0.1 as the best value on the Video-MME benchmark, which is tuning on an evaluation set.
  • video frames = 16 for LLaVA models, 32 for Qwen models
    Number of input frames is a training choice constrained by GPU memory and affects video understanding performance.
assumptions (5)
  • domain assumption Bradley-Terry model for preference pairs (Eq. 4)
    The paper assumes pairwise preferences follow the Bradley-Terry model with a logistic sigmoid, a standard choice in DPO/SimPO but not derived for video responses.
  • ad hoc to paper Average log-likelihood is a valid reward for preference optimization
    The central reward reformulation in Eq. 2-3 assumes likelihood correlates with response quality, yet Fig. 3 shows ground-truth answers have the lowest likelihood.
  • domain assumption Injecting the ground-truth answer as a hint and reflecting improves the winning response
    The data pipeline in Sec. 3.1 relies on this to produce high-trustworthiness, high-likelihood winning responses.
  • domain assumption Video augmentation degrades response quality enough to create valid losing responses
    The pipeline assumes augmented videos lead to lower-quality responses, which is plausible but not verified.
  • domain assumption Training data and evaluation benchmarks do not overlap
    No contamination analysis is provided, so the reported gains assume the 17k LLaVA-Hound samples are disjoint from Video-MME, MLVU, NeXT-QA, and other benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs." pith.science (2026). https://pith.science/paper/P5I3RYXP

@misc{pith2026250605260,
  author       = {Pith},
  title        = {Pith review of: LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P5I3RYXP}},
  note         = {Machine review of arXiv:2506.05260}
}
abstract

Most Video Large Language Models (Video-LLMs) adopt preference alignment techniques, e.g., DPO~\citep{rafailov2024dpo}, to optimize the reward margin between a winning response ($y_w$) and a losing response ($y_l$). However, the likelihood displacement observed in DPO indicates that both $\log \pi_\theta (y_w\mid x)$ and $\log \pi_\theta (y_l\mid x) $ often decrease during training, inadvertently boosting the probabilities of non-target responses. In this paper, we systematically revisit this phenomenon from LLMs to Video-LLMs, showing that it intensifies when dealing with the redundant complexity of video content. To alleviate the impact of this phenomenon, we propose \emph{Lean Preference Optimization} (LeanPO), a reference-free approach that reformulates the implicit reward as the average likelihood of the response with respect to the policy model. A key component of LeanPO is the reward-trustworthiness correlated self-generated preference data pipeline, which carefully infuses relevant prior knowledge into the model while continuously refining the preference data via self-reflection. This allows the policy model to obtain high-quality paired data and accurately estimate the newly defined reward, thus mitigating the unintended drop. In addition, we introduce a dynamic label smoothing strategy that mitigates the impact of noise in responses from diverse video content, preventing the model from overfitting to spurious details. Extensive experiments demonstrate that LeanPO significantly enhances the performance of state-of-the-art Video-LLMs, consistently boosting baselines of varying capacities with minimal additional training overhead. Moreover, LeanPO offers a simple yet effective solution for aligning Video-LLM preferences with human trustworthiness, paving the way toward the reliable and efficient Video-LLMs.

Figures

Figures reproduced from arXiv: 2506.05260 by the authors.

Figure 1
Figure 1. (a). Comparison between a LLM alignment based on Llama3-8B-Instruct [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the LEANPO (Lean Preference Optimization) framework. (1) Given an input video, the system prompt, and the user query, a novel method that injects the prior (the ground-truth answer with high trustworthiness) into the Video-LLM to generate the winning response, which captures the trustworthiness from the standard answer, and an losing response is generated by augmented video without prior injection. (2) P… view at source ↗
Figure 3
Figure 3. Implicit rewards for different responses. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison for Video Detail Captioning and Video QA. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Impact of self-generated preference dataset size based on LLaVA-NeXT-Video-7B [6] for on Video-MME. The present results are in the format “w.o subs". The model only traverses the preference dataset once during training. The intuitive result is that increasing the data …
Figure 6
Figure 6. Figure 6: Visualization of average likelihood (implicit reward) for each video sample with different Winning response Video-LLM Augmentation [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Comparison of different methods for Video-LLM alignment. We compare the preference [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: More Video QA results compared with baseline. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: More Video detailed description results compared with baseline. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: More Video detailed description results compared with baseline. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 14 canonical work pages

  1. [61]

    Temporal alignment networks for long-term video

    Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal alignment networks for long-term video. InCVPR, pages 2906–2916, 2022

    work page 2022

  2. [1]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NIPS, 36, 2024. 10

    work page 2024

  3. [2]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. InCVPR, pages 14398–14409, 2024

    work page 2024

  4. [3]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. InCVPR, pages 26689–26699, 2024

    2024

  5. [4]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR, pages 26296–26306, 2024

    2024

  6. [5]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  7. [6]

    Llava-next: A strong zero-shot video understanding model, April 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024

    2024

  8. [7]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    arXiv 2024

Show all 72 references
  1. [8]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  2. [9]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024

    2024

  3. [10]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. InAAAI, volume 38, pages 18135–18143, 2024

    2024

  4. [11]

    Tuning large multimodal models for videos using reinforcement learning from AI feedback

    Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. Tuning large multimodal models for videos using reinforcement learning from AI feedback. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association f...

    2024

  5. [12]

    i-srt: Aligning large multimodal models for videos by iterative self-retrospective judgment.arXiv preprint arXiv:2406.11280, 2024

    Daechul Ahn, Yura Choi, San Kim, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. i-srt: Aligning large multimodal models for videos by iterative self-retrospective judgment.arXiv preprint arXiv:2406.11280, 2024

    2024 arXiv

  6. [13]

    Direct preference optimization of video large multimodal models from language model reward.arXiv preprint arXiv:2404.01258, 2024

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward.arXiv preprint arXiv:2404.01258, 2024

    2024 arXiv

  7. [14]

    Temporal preference optimization for long-form video understanding.arXiv preprint arXiv:2501.13919, 2025

    Rui Li, Xiaohan Wang, Yuhui Zhang, Zeyu Wang, and Serena Yeung-Levy. Temporal preference optimization for long-form video understanding.arXiv preprint arXiv:2501.13919, 2025

    2025 arXiv

  8. [17]

    Unintentional unalignment: Likelihood displacement in direct preference optimization.arXiv preprint arXiv:2410.08847, 2024

    Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization.arXiv preprint arXiv:2410.08847, 2024

    2024 arXiv

  9. [18]

    From r to q∗: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358, 2024

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q∗: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358, 2024

    2024 arXiv

  10. [19]

    Smaug: Fixing failure modes of preference optimisation with dpo-positive.arXiv preprint arXiv:2402.13228, 2024

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive.arXiv preprint arXiv:2402.13228, 2024. 11

    2024 arXiv

  11. [20]

    Preference fine-tuning of llms should leverage suboptimal, on-policy data.arXiv preprint arXiv:2404.14367, 2024

    Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data.arXiv preprint arXiv:2404.14367, 2024

    2024 arXiv

  12. [21]

    Dpo-shift: Shifting the distribution of direct preference optimization.arXiv preprint arXiv:2502.07599, 2025

    Xiliang Yang, Feng Jiang, Qianen Zhang, Lei Zhao, and Xiao Li. Dpo-shift: Shifting the distribution of direct preference optimization.arXiv preprint arXiv:2502.07599, 2025

    2025 arXiv

  13. [22]

    Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer.arXiv preprint arXiv:2405.16436, 2024

    Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer.arXiv preprint arXiv:2405.16436, 2024

    2024 arXiv

  14. [23]

    Introducing chatgpt.CoRR, 2022

    OpenAI. Introducing chatgpt.CoRR, 2022

    2022

  15. [24]

    GPT-4 technical report.CoRR, 2023

    OpenAI. GPT-4 technical report.CoRR, 2023

    2023

  16. [25]

    LLaMA 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. LLaMA 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  17. [26]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    Qwen team. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  18. [27]

    Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    2017

  19. [28]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, , et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  20. [29]

    Preference ranking optimization for human alignment

    Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18990–18998, 2024

    2024

  21. [30]

    Lipo: Listwise preference optimization through learning-to-rank.arXiv preprint arXiv:2402.01878, 2024

    Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al. Lipo: Listwise preference optimization through learning-to-rank.arXiv preprint arXiv:2402.01878, 2024

    2024 arXiv

  22. [31]

    Orpo: Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189, 2024

    2024

  23. [32]

    Simpo: Simple preference optimization with a reference-free reward.arXiv preprint arXiv:2405.14734, 2024

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward.arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  24. [33]

    Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306, 2024

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  25. [34]

    β-dpo: Direct preference optimization with dynamic β.Advances in Neural Information Processing Systems, 37:129944–129966, 2024

    Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. β-dpo: Direct preference optimization with dynamic β.Advances in Neural Information Processing Systems, 37:129944–129966, 2024

    2024

  26. [35]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. InCVPR, pages 13807–13816, 2024

    2024

  27. [36]

    Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness.arXiv preprint arXiv:2405.17220, 2024

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness.arXiv preprint arXiv:2405.17220, 2024

    2024

  28. [37]

    Silkie: Preference distillation for large visual language models

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665, 2023

    2023 arXiv

  29. [38]

    Calibrated self-rewarding vision language models.arXiv preprint arXiv:2405.14622, 2024

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models.arXiv preprint arXiv:2405.14622, 2024. 12

    2024 arXiv

  30. [39]

    Aligning modali- ties in vision large language models via preference fine-tuning.arXiv preprint arXiv:2402.11411, 2024

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modali- ties in vision large language models via preference fine-tuning.arXiv preprint arXiv:2402.11411, 2024

    2024 arXiv

  31. [40]

    Strengthening multimodal large language model with bootstrapped preference optimization

    Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization. In European Conference on Computer Vision, pages 382–398. Springer, 2025

    2025

  32. [41]

    Self-supervised visual preference alignment.arXiv preprint arXiv:2404.10501, 2024

    Ke Zhu, Liang Zhao, Zheng Ge, and Xiangyu Zhang. Self-supervised visual preference alignment.arXiv preprint arXiv:2404.10501, 2024

    2024 arXiv

  33. [42]

    Enhancing large vision language models with self-training on image comprehension

    Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self-training on image comprehension. arXiv preprint arXiv:2405.19716, 2024

    2024 arXiv

  34. [43]

    V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization.arXiv preprint arXiv:2411.02712, 2024

    Yuxi Xie, Guanzhen Li, Xiao Xu, and Min-Yen Kan. V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization.arXiv preprint arXiv:2411.02712, 2024

    2024 arXiv

  35. [44]

    Llava-critic: Learning to evaluate multimodal models.arXiv preprint arXiv:2410.02712, 2024

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models.arXiv preprint arXiv:2410.02712, 2024

    2024 arXiv

  36. [45]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. https://cdn.openai.com/papers/GPTV_System_ Card.pdf, 2023

    2023

  37. [46]

    Mia-dpo: Multi-image augmented direct preference optimization for large vision-language models.arXiv preprint arXiv:2410.17637, 2024

    Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Mia-dpo: Multi-image augmented direct preference optimization for large vision-language models.arXiv preprint arXiv:2410.17637, 2024

    arXiv

  38. [47]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  39. [48]

    Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024

    2024

  40. [49]

    Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  41. [50]

    Next-qa: Next phase of question- answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. InCVPR, pages 9777–9786, 2021

    2021

  42. [51]

    Tarsier: Recipes for training and evaluating large video description models.arXiv preprint arXiv:2407.00634, 2024

    Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models.arXiv preprint arXiv:2407.00634, 2024

    2024 arXiv

  43. [52]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  44. [53]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  45. [54]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  46. [55]

    Sharegpt4video: Improving video understanding and generation with better captions.arXiv preprint arXiv:2406.04325, 2024

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video understanding and generation with better captions.arXiv preprint arXiv:...

    2024 arXiv

  47. [56]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF Conference on Computer Visi...

    2024

  48. [58]

    aws-prototyping/long-llava-qwen2-7b, 2024

    Yin Song and Chen Wu and Eden Duthie. aws-prototyping/long-llava-qwen2-7b, 2024

    2024

  49. [59]

    Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos.arXiv preprint arXiv:2408.14023, 2024

    Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos.arXiv preprint arXiv:2408.14023, 2024

    2024 arXiv

  50. [60]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  51. [62]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. arXiv preprint arXiv:2311.17005, 2023

    2023 arXiv

  52. [63]

    Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens.arXiv preprint arXiv:2404.03413, 2024

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens.arXiv preprint arXiv:2404.03413, 2024

    2024 arXiv

  53. [64]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning.arXiv preprint arXiv:2404.16994, 2024

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning.arXiv preprint arXiv:2404.16994, 2024

    2024 arXiv

  54. [65]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  55. [66]

    Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens

    Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens. InCVPR, pages 13151–13160, 2024

    2024

  56. [67]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. InCVPR, pages 18221–18232, 2024

    2024

  57. [68]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311.08046, 2023

    2023 arXiv

  58. [69]

    Cat: enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios.arXiv preprint arXiv:2403.04640, 2024

    Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, and Xiaochun Cao. Cat: enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios.arXiv preprint arXiv:2403.04640, 2024

    2024 arXiv

  59. [70]

    St-llm: Large language models are effective temporal learners.arXiv preprint arXiv:2404.00308, 2024

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners.arXiv preprint arXiv:2404.00308, 2024

    2024 arXiv

  60. [71]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  61. [72]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 14

    2025 arXiv

  62. [73]

    Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  63. [74]

    Enhancing visual-language modality alignment in large vision language models via self-improvement.arXiv preprint arXiv:2405.15973, 2024

    Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, et al. Enhancing visual-language modality alignment in large vision language models via self-improvement.arXiv preprint arXiv:2405.15973, 2024

    2024 arXiv

  64. [75]

    Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023. 15 A Dataset Details 16 B Visualiza...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.