{"id":"52d84762-b02b-43cd-987d-ae2b524e77a1","arxiv_id":"2411.16201","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A new 24k video VQA preference dataset and an iterative DPO-with-extrapolation training framework improve video MLLM alignment scores, but the improvements are measured with the same GPT-4o judge that labeled the dataset.","lead":"This paper builds a new video question-answering preference dataset by sampling answers from several video AI models and scoring them with GPT-4o, then trains a video AI with an iterative preference-learning method. It claims to fix the scarcity of high-quality video preference data and to improve alignment, but its evaluation uses the same GPT-4o judge that created the training labels.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on GPT-4o as both preference labeler and evaluator; without human validation, reported gains may reflect reward hacking.","rationale":"The paper's central empirical claim is that MMAIP-V improves preference learning and that Iter-W2S-RLAIF fully exploits it. For this claim to be credible, the outcome measure must reflect genuine VQA alignment. The only outcome measure in the main experiments is GPT-4o scoring (Fig. 9). The same GPT-4o model was used to create the preference pairs (Eq. 1, Fig. 10). This creates a circularity: DPO training with GPT-4o-derived labels should, if successful, increase GPT-4o's scores by construction. The reported improvements (ΔScore +0.28 on WebVid, etc.) are therefore consistent with the model learning to satisfy this specific judge rather than with improved video understanding. The paper anticipates the ground-truth-matching bias and argues that their video-conditioned evaluation is better, but it supplies no validation that GPT-4o's judgments align with human judgments. Without a human study or independent benchmark, the 'unbiased' claim is unsupported. Additional evidence of a confound: chosen responses are systematically longer than rejected responses (Appendix A.2), and GPT-4o is known to favor verbose answers; this makes length and style plausible alternative explanations. The ablations in Section 6.3 show that the dataset and iterative training help under the same GPT-4o judge, but they do not break the circularity. The paper does provide useful engineering contributions (dataset, code, ablation design), and the analysis of groundtruth unreliability is interesting, but these do not establish the central claim. Therefore the reader's REJECT verdict remains appropriate; the concern is load-bearing and unaddressed.","tokens_in":18069,"tokens_out":4760,"duration_ms":184021,"concrete_test":"Sample 300 responses from LLaVA-Hound-SFT, LLaVA-Hound-DPO, and Iter-W2S-RLAIF across WebVid and TGIF. Have three human annotators blind to model identity score each response on the same rubric used in Fig. 9 (relevance, accuracy, clarity, completeness, 1-5). Compute mean human scores and inter-annotator agreement, then compare the human preference gap between Iter-W2S-RLAIF and LLaVA-Hound-SFT against the GPT-4o gap in Tables 2 and 3. If humans do not rank Iter-W2S-RLAIF above SFT by a similar margin (e.g., at least 0.2 points on average), the reported improvements are likely artifacts of GPT-4o self-preference rather than genuine alignment gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 constructs MMAIP-V by scoring candidate responses with GPT-4o (Eq. 1) and selecting the highest/lowest as chosen/rejected; Section 5 and Tables 2-3 then evaluate trained models with GPT-4o using a closely related rubric (Fig. 9 vs. Fig. 10). The same model thus provides both the training signal and the outcome measure. Since DPO explicitly optimizes the policy to increase the implicit reward that GPT-4o scores approximate, an increase in GPT-4o score is a weak test of improved video understanding: the model may simply learn to produce longer, more detailed-sounding answers. Appendix A.2 shows chosen responses are longer than rejected ones (mean 298 vs. 253 characters), a known feature LLM judges spuriously prefer. The paper calls the evaluation 'unbiased' but provides no evidence: no human study, no correlation with human preference, no independent benchmark. The observation that LLaVA-Hound-DPO's margin over SFT shrinks from 4.91% to 1.11% under the new judge shows the judge changes rankings, but without a ground truth for 'better' we cannot know whether the new rankings are more correct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes MMAIP-V, a 24k video VQA preference dataset built by sampling responses from four video MLLMs and scoring them with GPT-4o, and Iter-W2S-RLAIF, an iterative DPO framework that updates the reference model and applies training-free parameter extrapolation. It also proposes a vision-grounded GPT-4o evaluation scheme. Experiments on three in-domain and four out-of-domain datasets report consistent gains over LLaVA-Hound-SFT, e.g., +0.28 on WebVid and +0.32 on MSRVTT on a 1-5 score. The central claim is that MMAIP-V is beneficial for preference learning and that Iter-W2S-RLAIF fully exploits its alignment information. The main weakness is that GPT-4o is used both to construct the preference labels (Section 3.1) and to evaluate the final performance (Sections 5 and 6), making the outcome measure circular.","tokens_in":18260,"tokens_out":6160,"duration_ms":57844,"significance":"If the claimed improvements reflect genuine gains in video understanding, the dataset construction pipeline and the iterative weak-to-strong training recipe would be a useful contribution to MLLM alignment. The manuscript provides code and dataset releases and includes ablations over dataset construction variants, which are valuable. However, the current evidence does not establish the central claim: there is no human validation, no independent objective benchmark, and no statistical significance assessment. The contribution is therefore conditional; its significance depends on whether the evaluation circularity can be resolved with external validation.","major_comments":[{"comment":"GPT-4o is used as the scoring function in Eq. (1) to select chosen and rejected responses and as the evaluator for all main results, with rubrics that substantially overlap (Fig. 10 vs Fig. 9). Because DPO is trained against the same preference signal that is later measured, the reported improvements (+0.20 to +0.32 on a 1-5 scale) are a weak test of improved video understanding and may reflect reward hacking, such as producing longer, more detailed-sounding answers: Appendix A.2 shows chosen responses average 298 characters versus 253 for rejected responses. The manuscript calls the evaluation \"unbiased\" but provides no human study, no correlation with human preference, and no independent benchmark. Please add external validation (e.g., human preference judgments or an objective benchmark) and report the judge's robustness to response length and style.","section":"§3.1, §5, Tables 2-3"},{"comment":"No error bars, confidence intervals, or multiple seeds are reported. The headline gains of 0.20-0.32 on the 1-5 scale and ratio gains of 2-5 percentage points could easily be within run-to-run noise. The ablations in Tables 4 and 7 show even smaller differences, e.g., WebVid score 4.71 vs 4.75 for H-DPO-17k versus MMAIP-V without iteration or extrapolation. Without repeated runs and significance tests, the empirical claims are not quantitatively supported. Please report means and standard deviations over at least three seeds and apply appropriate statistical tests.","section":"§6.1, Tables 2-3"},{"comment":"The hyperparameter α for parameter extrapolation was chosen \"based on the WebVid validation datasets,\" and then WebVid results are reported in Table 2. It is unclear whether this validation set is disjoint from the WebVid test set used in the main evaluation. If the validation set is not held out, the headline WebVid improvement is inflated by hyperparameter selection. Please clarify the split and, if necessary, use a nested validation procedure or report results for all α values.","section":"§6.1, Appendix B.3"},{"comment":"The argument that the proposed evaluation is more \"unbiased\" because it changes the ranking of LLaVA-Hound-DPO relative to LLaVA-Hound-SFT is not sufficient. Changing the evaluation judge changes the rankings, but without a ground-truth human judgment there is no evidence that the new rankings are more correct. To substantiate the claim that the previous evaluation is biased and the proposed one is unbiased, the manuscript should compare both judges against human judgments or another validated benchmark.","section":"§6.2"}],"minor_comments":[{"comment":"There are several typos, including \"Howerver\" in the introduction and \"vedio\" and \"serval\" in Section 2; the manuscript should be proofread.","section":"§1"},{"comment":"The second line of Eq. (1) should define y_l as the arg min; as written it repeats y_w = arg max.","section":"§3.1, Eq. (1)"},{"comment":"The text states that the TGIF ratio improves by 3.26%, but Table 3 reports a difference of 96.74 - 94.60 = 2.14 percentage points; this inconsistency should be corrected.","section":"§6.2"},{"comment":"The layout of Table 4 is confusing: the \"Base Model\" column appears to contain both model and data labels, and the row labels are not self-explanatory. Please restructure the table with clear row and column headers.","section":"Table 4"},{"comment":"The legend labels \"EXPO in stage1\" and \"DPO stage1 w/ EXPO\" are difficult to interpret; please label all curves and axes explicitly.","section":"Figure 7"},{"comment":"The claim that longer chosen responses are a risk of hallucination that is \"effectively mitigated using a scoring function\" is plausible but unsupported; please provide an analysis or rephrase the claim more cautiously.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The core problem is the evaluator circularity: GPT-4o provides both the training signal and the outcome measure. If the authors can add a human preference study or independent benchmarks and report statistical significance, the paper could be viable; without such external validation, I would lean toward rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper builds a 24k video VQA preference dataset (MMAIP-V) by sampling responses from four MLLMs and scoring them with GPT-4o, then trains with iterative DPO plus parameter extrapolation. The dataset and training recipe are real contributions: video preference data is scarce, and the ablations (scoring vs. prior capability vs. random pairing) give a clean picture of what helps. That part is solid engineering.\n\nThe problem is evaluation. GPT-4o scores the candidate responses to build preference pairs (Section 3.1), and the same GPT-4o, with a very similar rubric, scores the final model outputs (Section 5). The model is literally trained to increase the reward that GPT-4o approximates, and then judged by that same approximation. So the reported gains of +0.2 to +0.3 on a 1-5 scale could just be the model learning to write longer, more detailed-sounding answers. The paper even shows chosen responses average 298 characters vs. 253 for rejected ones — a known trigger for LLM judge bias. Calling this evaluation \"unbiased\" without a human study or any external benchmark is not defensible.\n\nThere are also smaller issues: no error bars or multiple seeds, and the gains are small. The observation that LLaVA-Hound-DPO's margin over SFT shrinks under the new judge shows judges change rankings, but it does not by itself show the new rankings are more correct.\n\nI don't think this is a reject-and-forget. The dataset and pipeline are likely useful to the video MLLM community, and the core problem is fixable: add human preference judgments on a sample, report agreement, evaluate on an existing benchmark with a different judge, and run at least three seeds. The paper deserves a serious referee, not a desk rejection — but the central empirical claim as written should be treated as unproven until those checks are done. My recommendation is to send it to review with the expectation of major revision.","headline":"A useful video preference dataset and training pipeline, but the central results rest on GPT-4o judging its own training signal, so the empirical claims need independent validation.","tokens_in":597,"tokens_out":629,"would_cite":false,"duration_ms":86263,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a fully automatic multi-AI pipeline can build high-quality video VQA preference data, and that iterative weak-to-strong DPO on that data lifts a 7B video-language model's alignment scores on seven test sets.","keywords":["video large language models","visual question answering","preference learning","direct preference optimization","reinforcement learning from AI feedback","dataset construction","weak-to-strong alignment","multimodal evaluation"],"falsifier":"Take a random sample of about 200 test questions from WebVid and TGIF, run LLaVA-Hound-SFT, LLaVA-Hound-DPO, and Iter-W2S-RLAIF on each, and have human annotators blind to model identity rank the answers for factual accuracy and completeness against the video; if the human rankings do not reproduce Iter-W2S-RLAIF's advantage, the reported score gains reflect conformity to GPT-4o's judgment rather than improved video understanding.","tokens_in":17797,"feed_emoji":"🎬","tokens_out":11773,"duration_ms":92343,"temperature":0.7,"pith_summary":"The paper tries to establish that high-quality video question-answering preference data can be produced automatically, without human annotation, and that a corresponding training scheme can turn that data into measurably better video-language models. It introduces MMAIP-V, a 24,000-pair dataset built by having four video-language models answer the same questions and using GPT-4o scores to keep the best and worst response per question, and Iter-W2S-RLAIF, which trains with two rounds of direct preference optimization while refreshing the reference model and extrapolating parameters along the alignment direction. Across three in-domain and four out-of-domain video VQA test sets, the trained 7B model scores higher than its SFT base and prior DPO baselines under the paper's video-grounded evaluation. If the claim holds, preference alignment for video-language models no longer depends on expensive or noisy manual annotation, since the same pipeline can generate fresh preference pairs on demand.","feed_headline":"Multi-AI feedback dataset lifts video VQA scores","feed_subtitle":"A 24k-pair dataset plus iterative weak-to-strong DPO raises WebVid scores from 4.50 to 4.78.","key_machinery":"The load-bearing object is MMAIP-V, a dataset of 24,000 preference triples (video, question, chosen answer, rejected answer), built from a 'zoo' of four video multimodal large language models — Qwen2-VL-72B-Instruct, Qwen2-VL-7B-Instruct, Qwen2-VL-2B-Instruct, and MiniCPM-V-2.6 — each sampled at temperature 1.0, with GPT-4o assigning integer scores from 1 to 5 on relevance, consistency, accuracy, specificity, comprehensiveness, and novel insight. The chosen response is the highest-scoring and the rejected response the lowest-scoring; video-question pairs where all responses tie are dropped. The training mechanism, Iter-W2S-RLAIF, applies the direct preference optimization loss in two stages on evenly split halves of the dataset, resets the reference model to the latest policy after each stage, and then performs parameter extrapolation, $\\theta^{*} = \\theta_{2} + \\alpha(\\theta_{2} - \\theta_{1})$, a training-free step that pushes the weight vector further along the alignment direction. The evaluation mechanism is a GPT-4o prompt that receives video frames together with the question and answer and scores relevance, accuracy, clarity, and completeness.","core_discovery":"On the paper's own terms, the central discovery is that sampling responses from a deliberately diverse collection of video multimodal large language models and using an external multimodal judge to score them yields preference pairs whose chosen answers are high-quality and whose rejected answers are genuinely varied, and that training on those pairs with iterative DPO plus parameter extrapolation extracts more alignment signal than a single DPO pass. Concretely, Iter-W2S-RLAIF raises the average evaluation score on WebVid from 4.50 for LLaVA-Hound-SFT to 4.78 and the ratio of answers scoring at least 3 from 93.52% to 97.67%, with similar gains on VIDAL, ActivityNet, SSV2, MSRVTT, MSVD, and TGIF. The paper further argues that the common evaluation scheme, which grades responses by how closely they match a ground-truth answer without seeing the video, is biased against the capabilities the training is meant to improve, and that its video-conditioned, multi-criteria evaluation is the fairer yardstick.","pith_inferences":["Editorial inference: a blind human head-to-head on the same test sets would show whether the reported gains are judged video understanding or conformity to the scoring model's preferences.","Editorial inference: the construction pipeline is not inherently video-specific; the same zoo-sampling-plus-scorer recipe could generate preference pairs for image, audio, or embodied tasks, and swapping GPT-4o for an open-weight judge would test whether the gains survive without a closed proprietary scorer.","Editorial inference: chosen answers come disproportionately from the 72B model and rejected answers from the 2B/7B models, so MMAIP-V may partly encode a model-capability prior rather than pure alignment information; ablations that hold model identity fixed while varying the scores would isolate the scoring signal."],"forward_implications":["If the central claim is correct, video VQA preference data can be produced automatically and cheaply, so the annotation bottleneck that currently limits MLLM alignment disappears for new video domains.","If correct, iterative reference-model updates plus parameter extrapolation add alignment gains on top of one-shot DPO, giving a training recipe that extracts more from the same dataset.","If correct, prior video-DPO improvements were inflated by ground-truth-matching evaluation: under the paper's video-conditioned evaluation, LLaVA-Hound-DPO's gain over its SFT base falls from roughly 5% to about 1%, so future evaluations should feed video frames to the judge.","If correct, the main dataset design choices are high-quality chosen responses and diverse rejected responses, since ablations that replace the scoring function with random or model-identity-based pairing degrade performance."],"supporting_citations":[{"why":"is the Qwen2-VL family of zoo models (72B, 7B, 2B) whose sampled responses populate MMAIP-V.","marker":"[2]"},{"why":"is the MiniCPM-V-2.6 zoo model that contributes high-scoring chosen responses.","marker":"[5]"},{"why":"supplies the VIDAL video-QA source and the LanguageBind-video encoder the base model uses.","marker":"[8]"},{"why":"provides the LLaVA-Hound-SFT base model and the LLaVA-Hound-DPO baseline plus its 17k preference dataset.","marker":"[9]"},{"why":"is the GPT-4o model used to score candidate responses during dataset construction and to grade test answers in the proposed evaluation.","marker":"[14]"},{"why":"introduces the direct preference optimization loss that Iter-W2S-RLAIF applies iteratively.","marker":"[16]"},{"why":"is the prior video RLAIF baseline that this work compares against and extends.","marker":"[21]"},{"why":"supplies the weak-to-strong parameter extrapolation rule applied after each DPO stage.","marker":"[23]"},{"why":"supplies the ActivityNet video-question pairs used both as a construction source and as an in-domain evaluation set.","marker":"[24]"},{"why":"supplies the WebVid video-question pairs used both as a construction source and as the primary in-domain evaluation set.","marker":"[25]"}],"fun_headline_variants":["Multi-AI feedback yields diverse video preference pairs","Iterative weak-to-strong training sharpens video VQA","MMAIP-V: 24k AI-scored pairs align video LLMs","Weak-to-strong DPO iteration lifts video answer scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire pipeline assumes GPT-4o is a fair and accurate judge: the paper itself says it uses GPT-4o 'due to the lack of a trained reliable reward model,' and GPT-4o both selects chosen and rejected responses while building MMAIP-V and grades every test answer in the main experiments, with no human evaluation to confirm those scores track real video-understanding quality.","fun_headline_variants_meta":{"raw":{"variants":["Multi-AI feedback yields diverse video preference pairs","Iterative weak-to-strong training sharpens video VQA","MMAIP-V: 24k AI-scored pairs align video LLMs","Weak-to-strong DPO iteration lifts video answer scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000455,"raw_usage":{"total_tokens":2379,"prompt_tokens":1134,"completion_tokens":1245,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":750,"completion_tokens_details":{"reasoning_tokens":1175}},"tokens_in":750,"tokens_out":1245,"duration_ms":9443,"temperature":1.0,"reasoning_tokens":1175,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:22:50.581433+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of about 200 test questions from WebVid and TGIF, run LLaVA-Hound-SFT, LLaVA-Hound-DPO, and Iter-W2S-RLAIF on each, and have human annotators blind to model identity rank the answers for factual accuracy and completeness against the video; if the human rankings do not reproduce Iter-W2S-RLAIF's advantage, the reported score gains reflect conformity to GPT-4o's judgment rather than improved video understanding.","supporting_citations":[],"review_version":1}