Pith. sign in

REVIEW 4 major objections 6 minor 47 references

Video-Text Dataset Construction from Multi-AI Feedback: Promoting Weak-to-Strong Preference Learning for Video Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a fully automatic multi-AI pipeline can build high-quality video VQA preference data, and that iterative weak-to-strong DPO on that data lifts a 7B video-language model's alignment scores on seven test sets.

desk verdict A useful video preference dataset and training pipeline, but the central results rest on GPT-4o judging its own training signal, so the empirical claims need independent validation. read the letter →

arxiv 2411.16201 v1 pith:SSZU5CXI submitted 2024-11-25 cs.LG cs.CLcs.CV

Hao Yi , Qingyang Li , Yulan Hu , Fuzheng Zhang , Di Zhang , Yong Liu This is my paper · ORCID
classification cs.LGcs.CLcs.CV
keywords videolargelanguagemodelsvisualquestionansweringpreferencelearningdirectoptimizationreinforcementfromAIfeedbackdatasetconstructionweak-to-strongalignmentmultimodalevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that high-quality video question-answering preference data can be produced automatically, without human annotation, and that a corresponding training scheme can turn that data into measurably better video-language models. It introduces MMAIP-V, a 24,000-pair dataset built by having four video-language models answer the same questions and using GPT-4o scores to keep the best and worst response per question, and Iter-W2S-RLAIF, which trains with two rounds of direct preference optimization while refreshing the reference model and extrapolating parameters along the alignment direction. Across three in-domain and four out-of-domain video VQA test sets, the trained 7B model scores higher than its SFT base and prior DPO baselines under the paper's video-grounded evaluation. If the claim holds, preference alignment for video-language models no longer depends on expensive or noisy manual annotation, since the same pipeline can generate fresh preference pairs on demand.

What carries the argument

The load-bearing object is MMAIP-V, a dataset of 24,000 preference triples (video, question, chosen answer, rejected answer), built from a 'zoo' of four video multimodal large language models — Qwen2-VL-72B-Instruct, Qwen2-VL-7B-Instruct, Qwen2-VL-2B-Instruct, and MiniCPM-V-2.6 — each sampled at temperature 1.0, with GPT-4o assigning integer scores from 1 to 5 on relevance, consistency, accuracy, specificity, comprehensiveness, and novel insight. The chosen response is the highest-scoring and the rejected response the lowest-scoring; video-question pairs where all responses tie are dropped. The training mechanism, Iter-W2S-RLAIF, applies the direct preference optimization loss in two stages on evenly split halves of the dataset, resets the reference model to the latest policy after each stage, and then performs parameter extrapolation, $\theta^{*} = \theta_{2} + \alpha(\theta_{2} - \theta_{1})$, a training-free step that pushes the weight vector further along the alignment direction. The evaluation mechanism is a GPT-4o prompt that receives video frames together with the question and answer and scores relevance, accuracy, clarity, and completeness.

What would settle it

Take a random sample of about 200 test questions from WebVid and TGIF, run LLaVA-Hound-SFT, LLaVA-Hound-DPO, and Iter-W2S-RLAIF on each, and have human annotators blind to model identity rank the answers for factual accuracy and completeness against the video; if the human rankings do not reproduce Iter-W2S-RLAIF's advantage, the reported score gains reflect conformity to GPT-4o's judgment rather than improved video understanding.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that sampling responses from a deliberately diverse collection of video multimodal large language models and using an external multimodal judge to score them yields preference pairs whose chosen answers are high-quality and whose rejected answers are genuinely varied, and that training on those pairs with iterative DPO plus parameter extrapolation extracts more alignment signal than a single DPO pass. Concretely, Iter-W2S-RLAIF raises the average evaluation score on WebVid from 4.50 for LLaVA-Hound-SFT to 4.78 and the ratio of answers scoring at least 3 from 93.52% to 97.67%, with similar gains on VIDAL, ActivityNet, SSV2, MSRVTT, MSVD, and TGIF. The paper further argues that the common evaluation scheme, which grades responses by how closely they match a ground-truth answer without seeing the video, is biased against the capabilities the training is meant to improve, and that its video-conditioned, multi-criteria evaluation is the fairer yardstick.

Load-bearing premise

The entire pipeline assumes GPT-4o is a fair and accurate judge: the paper itself says it uses GPT-4o 'due to the lack of a trained reliable reward model,' and GPT-4o both selects chosen and rejected responses while building MMAIP-V and grades every test answer in the main experiments, with no human evaluation to confirm those scores track real video-understanding quality.

Editorial extensions

If this is right

  • If the central claim is correct, video VQA preference data can be produced automatically and cheaply, so the annotation bottleneck that currently limits MLLM alignment disappears for new video domains.
  • If correct, iterative reference-model updates plus parameter extrapolation add alignment gains on top of one-shot DPO, giving a training recipe that extracts more from the same dataset.
  • If correct, prior video-DPO improvements were inflated by ground-truth-matching evaluation: under the paper's video-conditioned evaluation, LLaVA-Hound-DPO's gain over its SFT base falls from roughly 5% to about 1%, so future evaluations should feed video frames to the judge.
  • If correct, the main dataset design choices are high-quality chosen responses and diverse rejected responses, since ablations that replace the scoring function with random or model-identity-based pairing degrade performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a blind human head-to-head on the same test sets would show whether the reported gains are judged video understanding or conformity to the scoring model's preferences.
  • Editorial inference: the construction pipeline is not inherently video-specific; the same zoo-sampling-plus-scorer recipe could generate preference pairs for image, audio, or embodied tasks, and swapping GPT-4o for an open-weight judge would test whether the gains survive without a closed proprietary scorer.
  • Editorial inference: chosen answers come disproportionately from the 72B model and rejected answers from the 2B/7B models, so MMAIP-V may partly encode a model-capability prior rather than pure alignment information; ablations that hold model identity fixed while varying the scores would isolate the scoring signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes MMAIP-V, a 24k video VQA preference dataset built by sampling responses from four video MLLMs and scoring them with GPT-4o, and Iter-W2S-RLAIF, an iterative DPO framework that updates the reference model and applies training-free parameter extrapolation. It also proposes a vision-grounded GPT-4o evaluation scheme. Experiments on three in-domain and four out-of-domain datasets report consistent gains over LLaVA-Hound-SFT, e.g., +0.28 on WebVid and +0.32 on MSRVTT on a 1-5 score. The central claim is that MMAIP-V is beneficial for preference learning and that Iter-W2S-RLAIF fully exploits its alignment information. The main weakness is that GPT-4o is used both to construct the preference labels (Section 3.1) and to evaluate the final performance (Sections 5 and 6), making the outcome measure circular.

Significance. If the claimed improvements reflect genuine gains in video understanding, the dataset construction pipeline and the iterative weak-to-strong training recipe would be a useful contribution to MLLM alignment. The manuscript provides code and dataset releases and includes ablations over dataset construction variants, which are valuable. However, the current evidence does not establish the central claim: there is no human validation, no independent objective benchmark, and no statistical significance assessment. The contribution is therefore conditional; its significance depends on whether the evaluation circularity can be resolved with external validation.

major comments (4)
  1. [§3.1, §5, Tables 2-3] GPT-4o is used as the scoring function in Eq. (1) to select chosen and rejected responses and as the evaluator for all main results, with rubrics that substantially overlap (Fig. 10 vs Fig. 9). Because DPO is trained against the same preference signal that is later measured, the reported improvements (+0.20 to +0.32 on a 1-5 scale) are a weak test of improved video understanding and may reflect reward hacking, such as producing longer, more detailed-sounding answers: Appendix A.2 shows chosen responses average 298 characters versus 253 for rejected responses. The manuscript calls the evaluation "unbiased" but provides no human study, no correlation with human preference, and no independent benchmark. Please add external validation (e.g., human preference judgments or an objective benchmark) and report the judge's robustness to response length and style.
  2. [§6.1, Tables 2-3] No error bars, confidence intervals, or multiple seeds are reported. The headline gains of 0.20-0.32 on the 1-5 scale and ratio gains of 2-5 percentage points could easily be within run-to-run noise. The ablations in Tables 4 and 7 show even smaller differences, e.g., WebVid score 4.71 vs 4.75 for H-DPO-17k versus MMAIP-V without iteration or extrapolation. Without repeated runs and significance tests, the empirical claims are not quantitatively supported. Please report means and standard deviations over at least three seeds and apply appropriate statistical tests.
  3. [§6.1, Appendix B.3] The hyperparameter α for parameter extrapolation was chosen "based on the WebVid validation datasets," and then WebVid results are reported in Table 2. It is unclear whether this validation set is disjoint from the WebVid test set used in the main evaluation. If the validation set is not held out, the headline WebVid improvement is inflated by hyperparameter selection. Please clarify the split and, if necessary, use a nested validation procedure or report results for all α values.
  4. [§6.2] The argument that the proposed evaluation is more "unbiased" because it changes the ranking of LLaVA-Hound-DPO relative to LLaVA-Hound-SFT is not sufficient. Changing the evaluation judge changes the rankings, but without a ground-truth human judgment there is no evidence that the new rankings are more correct. To substantiate the claim that the previous evaluation is biased and the proposed one is unbiased, the manuscript should compare both judges against human judgments or another validated benchmark.
minor comments (6)
  1. [§1] There are several typos, including "Howerver" in the introduction and "vedio" and "serval" in Section 2; the manuscript should be proofread.
  2. [§3.1, Eq. (1)] The second line of Eq. (1) should define y_l as the arg min; as written it repeats y_w = arg max.
  3. [§6.2] The text states that the TGIF ratio improves by 3.26%, but Table 3 reports a difference of 96.74 - 94.60 = 2.14 percentage points; this inconsistency should be corrected.
  4. [Table 4] The layout of Table 4 is confusing: the "Base Model" column appears to contain both model and data labels, and the row labels are not self-explanatory. Please restructure the table with clear row and column headers.
  5. [Figure 7] The legend labels "EXPO in stage1" and "DPO stage1 w/ EXPO" are difficult to interpret; please label all curves and axes explicitly.
  6. [Appendix A.2] The claim that longer chosen responses are a risk of hallucination that is "effectively mitigated using a scoring function" is plausible but unsupported; please provide an analysis or rephrase the claim more cautiously.

Circularity Check

1 steps flagged · score 7.0 of 10

GPT-4o serves as both the preference labeler that builds MMAIP-V and the evaluator that produces the headline scores, so the central empirical claim largely measures the policy's fit to GPT-4o rather than independent alignment.

  1. fitted input called prediction [Section 3.1, Eq. (1); Section 5, evaluation prompts (Fig. 9 vs. Fig. 10)]
    "Due to the lack of trained reliable reward model, we utilize GPT-4o to score responses in a text-to-text manner. ... yw = arg max y∼M R(y; x, v), yl = arg min y∼M R(y; x, v) (Eq. 1). ... we introduce the video modality and perform a fine-grained evaluation of the inference answer quality by utilizing GPT-4o."

    The preference pairs in MMAIP-V are literally defined by GPT-4o scores: chosen responses are argmax over R and rejected responses are argmin over R, where R is GPT-4o. DPO then trains the policy to increase the implicit reward margin between those chosen and rejected responses. The main results in Tables 2 and 3 are then measured by GPT-4o scores using a closely related rubric. Thus the reported improvement is, by construction, an improvement on the same judge used to create the training signal; it does not provide independent evidence of better video understanding. The paper provides no human study, no correlation with human preference, and no external benchmark to anchor GPT-4o's scores. The appendix even shows chosen responses are on average longer than rejected ones (298 vs.

full rationale

This paper does not rely on a load-bearing self-citation chain; its references to LLaVA-Hound, VLM-RLAIF, and weak-to-strong extrapolation are external and independently published. The circularity is instead in the evaluation protocol: the same GPT-4o model both defines the preference signal in the training data (Eq. 1) and scores the final model outputs in the headline experiments (Section 5, Tables 2-3). Since DPO optimizes toward exactly the kind of preferences GPT-4o encodes, the reported gains can partly or largely reflect the policy learning to imitate the judge rather than a validated improvement in video QA quality. This is a partial but central circularity: the dataset's quality and the framework's benefit are both measured by the AI system that generated the labels, with no human ground truth. The paper is coherent as a recipe for aligning to GPT-4o feedback, but its stronger claims of unbiased evaluation and hallucination reduction would need an independent judge or human study. I therefore assign a score of 7: significant evaluator-reward overlap makes the central validation largely self-confirming, though not a pure tautology.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the assumption that GPT-4o is a valid judge for both data construction and evaluation, plus the transfer of parameter extrapolation to MLLMs. The free parameters are tuning choices (α and β) that affect the final model but are standard practice. No new physical or conceptual entities are introduced.

free parameters (2)
  • alpha (α) extrapolation strength = 0.5 in stage 1 (best on WebVid validation); stage 2 optimal not explicitly stated
    Tuned on WebVid validation set (Appendix B.3, D.1); directly controls the parameter extrapolation step in Eq. (4) and therefore affects the final model weights.
  • beta (β) in DPO loss = 0.1
    Set by hand to 0.1 (Appendix B.3); standard DPO hyperparameter controlling the implicit reward scale, affects the DPO loss in Eq. (2).
assumptions (4)
  • domain assumption GPT-4o scores are a reliable, unbiased measure of VQA response quality
    Used as reward for dataset construction (Section 3.1) and as evaluation metric (Section 5); no external validation of this judge is provided.
  • domain assumption Ground truth in existing video VQA benchmarks is often incomplete or wrong, so matching ground truth is a biased evaluation
    Section 5 argues this based on examples and score comparisons (Figure 3), but the comparisons themselves are GPT-4o judgments.
  • ad hoc to paper Parameter extrapolation of the weight vector increases alignment, i.e., θ2 + α(θ2 - θ1) is better aligned
    From prior work [23] and assumed to transfer to MLLMs; the paper does not derive this but relies on it for Iter-W2S-RLAIF (Eq. 4).
  • standard math Bradley-Terry preference model and DPO loss
    Equation (2) uses DPO, a standard result; this is a background assumption for the training objective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Video-Text Dataset Construction from Multi-AI Feedback: Promoting Weak-to-Strong Preference Learning for Video Large Language Models." pith.science (2026). https://pith.science/paper/SSZU5CXI

@misc{pith2026241116201,
  author       = {Pith},
  title        = {Pith review of: Video-Text Dataset Construction from Multi-AI Feedback: Promoting Weak-to-Strong Preference Learning for Video Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SSZU5CXI}},
  note         = {Machine review of arXiv:2411.16201}
}
read the original abstract

High-quality video-text preference data is crucial for Multimodal Large Language Models (MLLMs) alignment. However, existing preference data is very scarce. Obtaining VQA preference data for preference training is costly, and manually annotating responses is highly unreliable, which could result in low-quality pairs. Meanwhile, AI-generated responses controlled by temperature adjustment lack diversity. To address these issues, we propose a high-quality VQA preference dataset, called \textit{\textbf{M}ultiple \textbf{M}ultimodal \textbf{A}rtificial \textbf{I}ntelligence \textbf{P}reference Datasets in \textbf{V}QA} (\textbf{MMAIP-V}), which is constructed by sampling from the response distribution set and using an external scoring function for response evaluation. Furthermore, to fully leverage the preference knowledge in MMAIP-V and ensure sufficient optimization, we propose \textit{\textbf{Iter}ative \textbf{W}eak-to-\textbf{S}trong \textbf{R}einforcement \textbf{L}earning from \textbf{AI} \textbf{F}eedback for video MLLMs} (\textbf{Iter-W2S-RLAIF}), a framework that gradually enhances MLLMs' alignment capabilities by iteratively updating the reference model and performing parameter extrapolation. Finally, we propose an unbiased and information-complete evaluation scheme in VQA evaluation. Experiments demonstrate that MMAIP-V is beneficial for MLLMs in preference learning and Iter-W2S-RLAIF fully exploits the alignment information in MMAIP-V. We believe that the proposed automatic VQA preference data generation pipeline based on AI feedback can greatly promote future work in the MLLMs alignment. \textbf{Code and dataset are available} \href{https://anonymous.4open.science/r/MMAIP-V_Iter-W2S-RLAIF-702F}{MMAIP-V\_Iter-W2S-RLAIF-702F}.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 8 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    arXiv 2024

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  4. [4]

    Llava-next- interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next- interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024

    arXiv 2024

  5. [5]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 10 A PREPRINT - NOVEMBER 26, 2024

    arXiv 2024

  6. [6]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023

    arXiv 2023

  7. [7]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023

    arXiv 2023

  8. [8]

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023

    arXiv

Show all 47 references
  1. [9]

    Direct preference optimization of video large multimodal models from language model reward

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01258, 2024

    2024 arXiv

  2. [10]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  3. [11]

    Aligning large multi-modal model with robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023

    2023 arXiv

  4. [12]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  5. [13]

    Aligning large multimodal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  6. [14]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  7. [15]

    Self-rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024

    2024 arXiv

  8. [16]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  9. [17]

    Statistical rejection sampling improves preference optimization

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657, 2023

    2023 arXiv

  10. [18]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  11. [19]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Co...

    2024

  12. [20]

    Silkie: Preference distillation for large visual language models

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665, 2023

    2023 arXiv

  13. [21]

    Tuning large multimodal models for videos using reinforcement learning from ai feedback

    Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. Tuning large multimodal models for videos using reinforcement learning from ai feedback. arXiv preprint arXiv:2402.03746, 2024

    2024 arXiv

  14. [22]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952

    1952

  15. [23]

    Weak-to-strong extrapolation expedites alignment

    Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Weak-to-strong extrapolation expedites alignment. arXiv preprint arXiv:2404.16792, 2024

    2024 arXiv

  16. [24]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015

    2015

  17. [25]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1728–1738, 2021

    2021

  18. [26]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2025. 11 A PREPRINT - NOVEMBER 26, 2024

    2025

  19. [27]

    Llava-next: A strong zero-shot video understanding model, April 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024

    2024

  20. [28]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  21. [29]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  22. [30]

    Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839, 2023

    2023 arXiv

  23. [31]

    Understanding reference policies in direct preference optimization

    Yixin Liu, Pengfei Liu, and Arman Cohan. Understanding reference policies in direct preference optimization. arXiv preprint arXiv:2407.13709, 2024

    2024 arXiv

  24. [32]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018

    2018 arXiv

  25. [33]

    Mitigating the alignment tax of rlhf

    Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, et al. Mitigating the alignment tax of rlhf. CoRR, 2023

    2023

  26. [34]

    Spurious feature diversification improves out-of-distribution generalization

    Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distribution generalization. arXiv preprint arXiv:2309.17230, 2023

    2023 arXiv

  27. [35]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  28. [36]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016

    2016

  29. [37]

    Collecting highly parallel data for paraphrase evaluation

    David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190–200, 2011

    2011

  30. [38]

    Tgif: A new dataset and benchmark on animated gif description

    Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4641–4650, 2016

    2016

  31. [39]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Pr...

    2017

  32. [40]

    Eva: Exploring the limits of masked visual representation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

    2023

  33. [41]

    Vision transformer with quadrangle attention

    Qiming Zhang, Jing Zhang, Yufei Xu, and Dacheng Tao. Vision transformer with quadrangle attention. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  34. [42]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  35. [43]

    % 4.47 95.30 4.45 91.63 4.60 94.39 4.71 96.08

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 12 A PREPRINT - NOVEMBER 26, 2024 0 200 400 600 800 1000 Responses Length/Word level 0.000 0.005 0.010 0.015 0.020 0.025 0.030 0.035Density chosen_responses_length rejected_responses_le...

    2017 arXiv

  36. [44]

    Video with Mutil-Frames #### Input into vision encoder

  37. [45]

    **Question Related to the Video**: {question}

  38. [46]

    Consider the following criteria for evaluation: -**Relevance**:Evaluate how relevant the model's predicted answer is to the question asked

    **Model Predicted Answer**: {prediction} Your task is to evaluate the model's predicted answer , based on the context provided by the video and the question. Consider the following criteria for evaluation: -**Relevance**:Evaluate how relevant the model's predicted answer is to...

    2024

  39. [47]

    Temporal Consistency:Does the answer appropriately reflect the temporal progression and events of the video?3

    Relevance to Visual Content:Does the candidate answer align with the visual evidence provided in the video frames?2. Temporal Consistency:Does the answer appropriately reflect the temporal progression and events of the video?3. Contextual Accuracy:Does the answer accurately ca...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.