Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reinforcement-learning post-training that rewards both correct answers and accurate timing lifts a video LLM above supervised fine-tuning on eight benchmarks while training on just 32,000 of 490,000 samples.

desk verdict Solid applied RLT recipe for video LLMs, but the data-efficiency claim needs a held-out threshold check before I'd trust it. read the letter →

arxiv 2506.01908 v1 pith:CVGHA6G4 submitted 2025-06-02 cs.CV

Hongyu Li , Songhao Han , Yue Liao , Junfeng Luo , Jialin Gao , Shuicheng Yan , Si Liu This is my paper
classification cs.CV
keywords reinforcementlearningtuningvideounderstandingtemporalgroundingquestionansweringGRPOrewarddesigndataselectionmultimodallargelanguagemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that reinforcement-learning tuning (RLT), not just supervised fine-tuning, is the right post-training step for video reasoning in multimodal LLMs, provided the reward signals match the two things video understanding demands: what happens and when it happens. Built on the Group Relative Policy Optimization (GRPO) algorithm, which learns by comparing several sampled outputs for the same input, Temporal-RLT combines a discrete reward for multi-choice video question-answering correctness with a continuous temporal-IoU reward for grounding, plus a format reward that enforces structured think-then-answer output. Its second claim is about data: because GRPO learns from the spread of rewards across sampled outputs, only samples whose outputs vary are informative, so the paper filters its 490k training pool by the base model's output variance and keeps the medium-difficulty middle, yielding a 32k subset. That subset beats supervised fine-tuning and the prior RLT baseline on all eight benchmarks, with temporal-grounding gains of about +14 mIoU over SFT on Charades-STA and ActivityNet and +9.5 on the reasoning-focused ActivityNet-RTL. A sympathetic reader would care because the recipe of task-matched rewards plus variance-based sample selection points to a way of making video models reason more accurately without collecting larger datasets.

What carries the argument

The load-bearing mechanism is the group-relative advantage of GRPO, $A_i = (R_i - \mathrm{mean}(\{R_1,\dots,R_G\})) / \mathrm{std}(\{R_1,\dots,R_G\})$, which converts the spread of rewards inside a sampled group into the training update. The paper's dual rewards feed that advantage: a discrete accuracy reward $R_{acc} \in \{0,1\}$ for multi-choice QA, a continuous temporal-IoU reward $R_{IoU}$ between predicted and ground-truth segments, and a format reward $R_{format}$ for emitting the structured `<think>`/`<answer>` (and `<observe>`) template, with grounded QA scored as $\frac{1}{2}(R_{acc} + R_{IoU})$. The companion mechanism is variance-aware sample selection by repeated inference: for QA, keep the samples whose correct-answer count lies in the middle band, and for grounding, keep samples with $\Delta IoU = \max_i IoU_i - \mathrm{mean}_i(IoU_i) \ge 0.3$. The connection between the two mechanisms is that both target the same quantity, reward variance within a group, once inside the optimizer and once at the dataset level, so the selected samples are exactly those that can still produce nonzero advantages.

What would settle it

Retrain the identical GRPO pipeline on a random 32k subset drawn from the same 490k pool and compare on Charades-STA, ActivityNet-RTL, and MMVU: if the random subset matches the variance-selected subset, the data-efficiency claim is refuted. A second check is to log the actual group advantages produced during training by samples labeled easy, medium, and hard; if hard-labeled samples systematically generate the largest advantages, the medium-only selection policy is discarding useful signal.

Watch

Extended reading notes

Core claim

The central claim is that GRPO-based post-training works for video understanding when each reasoning dimension receives its own verifiable reward and when training data is chosen for its capacity to generate preference signal rather than for diversity alone. For multi-choice video QA, the reward is binary answer correctness; for temporal grounding, it is the temporal IoU between the predicted and ground-truth intervals; grounded QA receives both, each at half weight. Under GRPO, each input is sampled multiple times and rewards are normalized within the group to form advantages, so a sample is useful exactly when its outputs span a wide reward range. The variance-aware selection operationalizes that: QA samples are labeled easy, medium, or hard by how many of the base model's repeated answers are correct, with the mixed medium band retained, and grounding samples are kept when the gap between maximum and mean IoU is at least 0.3. From a 490k pool this yields a 32k training set, and the resulting Temporal-RLT model outperforms supervised fine-tuning and the earlier RLT baseline on all eight benchmarks, with the largest gains in temporal grounding, while avoiding the general-VideoQA degradation that SFT produces.

Load-bearing premise

The result rests on the premise that a sample's usefulness for GRPO training is reliably measured by the base model's output variance, namely the count of correct answers for QA and the max-minus-mean IoU spread for grounding, with the difficulty cutoffs tuned on the same benchmarks later used to score success, so on models, domains, or data pools where variance does not track learning signal, the 32k-sample gains would not be expected to transfer.

Editorial extensions

If this is right

  • Temporal grounding improves by double-digit mIoU margins over SFT: +14.0 on Charades-STA, +14.7 on ActivityNet, and +9.5 on the reasoning-focused ActivityNet-RTL.
  • RL tuning avoids the generalization cost of SFT: where SFT drops general VideoQA scores (MVBench falls from 65.3 to 62.0), Temporal-RLT raises them (MVBench 68.1, MMVU 65.0 versus 52.5 for SFT).
  • Data efficiency is real at the stated scale: the 32k refined subset outperforms the prior RLT baseline even though it is about 15 times smaller than the full 490k pool, indicating sample informativeness can substitute for data volume in GRPO training.
  • The RL-tuned model benefits more than the SFT model from more video tokens (raising 2048 to 4096 tokens lifts Charades mIoU from 57.0 to 58.0 and MMVU from 65.0 to 66.1), so the gains compound with better temporal encoding.
  • Training on a single grounding dataset transfers out of domain: +16.0 mIoU on ActivityNet and +8.3 on ActivityNet-RTL from Charades-only training, whereas SFT transfer is small or negative, suggesting RLT imparts a transferable reasoning style.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The variance filter is effectively a near-decision-boundary selector: medium-difficulty QA samples and high-delta-IoU grounding samples sit where the model's outputs straddle correctness. The same rule could therefore run as a closed-loop curriculum, re-estimating difficulty during training to keep selecting the shrinking informative band, which the paper's own distribution-shift analysis motivate
  • Because the selection rule needs only black-box repeated inference, it is independent of the reward functions and should transfer to other verifiable rewards and other base models; re-deriving the 0.3 and 1/7 cutoffs for a different model or domain would test how universal those thresholds are.
  • The paper's ablations vary the difficulty mix but do not include a random 32k-subset control drawn from the same 490k pool, so how much of the gain comes from the variance filter versus from RL training at any scale remains an open quantity; a random-subset comparison would isolate the selection mechanism's contribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Temporal-RLT, a GRPO-based reinforcement learning tuning framework for VideoLLMs. It introduces a dual-reward formulation: a discrete accuracy reward for multi-choice video QA and a continuous temporal IoU reward for temporal grounding, plus a variance-aware data selection strategy that uses repeated inference to identify training samples likely to produce informative preference gradients. Experiments on Qwen-VL-2.5-7B across eight benchmarks (general VideoQA, reasoning VideoQA, temporal grounding, grounded VideoQA) report consistent improvements over SFT and Video-R1 baselines, with the best results obtained using a 32k-sample subset selected from a 490k pool.

Significance. If the reported results are reliable, the paper makes a useful contribution by showing that GRPO with task-specific verifiable rewards can improve both semantic and temporal video reasoning, and that difficulty-aware data selection can substantially reduce training data requirements. The dual-reward design is simple and intuitive, and the variance-based selection idea is a plausible mechanism for identifying useful GRPO training samples. However, the significance is currently undermined by an evaluation protocol in which the data selection thresholds are tuned on the same benchmarks used for final reporting, and by the absence of statistical significance measures and a second base model. The core idea is worth publishing after a rigorous re-evaluation with a properly held-out validation split and additional control experiments.

major comments (4)
  1. [Section 4.3 / Appendix A (Tables 4a, 4b and Table 1)] The data selection thresholds are tuned on benchmarks that also appear in the main results. The ablation in Table 4a evaluates Easy/Middle/Hard proportions on MVBench, TempCompass, VideoMME, and MMVU, and Table 4b evaluates ΔIoU thresholds on Charades-STA; all five of these benchmarks appear in Table 1's final evaluation table. Since the 32k subset is constructed using thresholds chosen to maximize performance on these exact benchmarks, the reported gains conflate genuine method improvements with benchmark-specific tuning, and the 'data efficiency' claim may not transfer to new distributions. The authors should use a held-out validation split (or a separate benchmark set) for threshold selection and report results on truly unseen benchmarks.
  2. [Section 3.2 / Appendix A] The definition of difficulty classes is internally inconsistent. The text states 'Easy if c≥τ_easy, Hard if c≤τ_hard, otherwise Medium' with τ_easy=1 and τ_hard=7. For N=8, this classification makes every sample with 1≤c≤7 both Easy and Hard, leaving no Medium samples, yet the ablation tables and the 16k 'medium' selection in Appendix A depend on a meaningful Medium category. The authors must clarify the intended thresholds (e.g., Easy if c≤τ_easy, Hard if c≥N−τ_hard+1, or a similar scheme) and ensure the notation is unambiguous; otherwise the reported data-selection procedure cannot be reproduced or verified.
  3. [Section 4.1 / Table 1] The comparison to Video-R1 is incomplete and the 'significantly less training data' claim is not quantified. Video-R1 cells are missing for the temporal grounding benchmarks and for NextGQA, so the statement that Temporal-RLT 'consistently outperforms' Video-R1 is only partially supported. Moreover, the paper does not report Video-R1's training data size or the compute budget used for that baseline, making the data-efficiency comparison unverifiable. The authors should provide a complete baseline comparison and explicitly state the training data sizes, number of steps, and computational cost for all methods.
  4. [Section 4.1 / Tables 1–5] All results are reported without error bars or multiple seeds, and only a single base model (Qwen-VL-2.5-7B) is used. Several reported differences are small (e.g., TempCompass 73.3 vs. 72.6 for Video-R1, VideoMME 57.6 vs. 57.4), and without variance estimates it is unclear whether these improvements are statistically meaningful. At least three seeds should be run and standard deviations reported; a second base model (or a smaller/larger variant) would also strengthen the generalization claims.
minor comments (6)
  1. [Abstract] The final sentence reads 'The latest version is available atTemporal-RLT.', which appears truncated and missing the actual URL or citation; it should be corrected to a proper link or reference.
  2. [Table 2] The entry for the '+w/ VQA' row under NextGQA reads '78.,4', which is a typo for '78.4'; the table should be carefully proofread.
  3. [Section 3.4 / Figure 3 caption] The caption asserts 'We find that converting all answers to an open-ended format is critical in reliably assessing question difficulty,' but this claim is not discussed anywhere in the main text or appendix; add supporting experimental evidence or remove the assertion.
  4. [Section 6 (Conclusion)] The conclusion refers to 'a dynamic sampling strategy that progressively refines the training set,' but the method described in Sections 3.4 and Appendix A is a one-time offline selection; the wording should be reconciled to avoid implying a dynamic procedure that is not actually described.
  5. [Section 2.3 (Equation 3)] The advantage normalization in Equation 3 uses 'mean' and 'std' without explicitly stating that these are computed over the group of G responses; this is standard GRPO but should be stated for self-containedness.
  6. [Related Work] The related work section would benefit from a brief discussion of how variance-based data selection for RL differs from standard difficulty-based curriculum learning (e.g., self-paced learning), to better position the contribution.

Circularity Check

1 steps flagged · score 4.0 of 10

Data-selection thresholds are tuned on the same benchmarks used for final evaluation, making the headline data-efficiency comparison a fitted result rather than an out-of-sample prediction.

  1. fitted input called prediction [Appendix A (Data Selection Implementation Details); Section 4.3 Tables 4a/4b; Section 4.2 Table 1]
    "Based on our ablation study, which showed that medium-level data performs best for multi-choice tasks, we select 16k samples from the medium category. ... For continuous rewards data, our ablation study revealed that a larger ∆IoU leads to better results. Therefore, we select data with ∆IoU ≥0.3 ."

    The thresholds τ_easy=1, τ_hard=7 and ΔIoU≥0.3 are justified by ablations run on MVBench, TempCompass, VideoMME, MMVU, and Charades-STA (Tables 4a/4b). Those are the same benchmarks used as final evaluation in Table 1. The final claim that Temporal-RLT-32k 'consistently outperforms' baselines with 'significantly less training data' is therefore not an independent test of the data-efficiency hypothesis: the subset composition was chosen to maximize scores on the very benchmarks where the result is reported. This is the fitted-input-called-prediction pattern: a selection hyperparameter fitted to the evaluation target is presented as evidence for a general data-efficiency gain.

full rationale

The paper is an empirical systems paper, and most of its proposed machinery is not circular: the dual discrete/continuous rewards are defined from ground-truth correctness and temporal IoU, and GRPO training with those rewards is evaluated on held-out test annotations rather than being derived from the training objective by construction. The variance-aware selection is also an empirical heuristic, not a tautology. The one substantive circularity concern is the selection of the data-filtering thresholds. Appendix A states that τ_easy=1, τ_hard=7 are set 'Based on our ablation study', and that the continuous-reward cutoff ΔIoU≥0.3 is likewise chosen because 'a larger ∆IoU leads to better results'. Those ablations (Tables 4a/4b) are run on MVBench, TempCompass, VideoMME, MMVU, and Charades-STA—the same benchmarks that later appear in the final main-results Table 1. Thus the reported 'superior performance with significantly less training data' is the output of a model-selection procedure that already maximized scores on the final evaluation benchmarks. This makes the headline data-efficiency comparison vulnerable to benchmark-specific tuning rather than a clean out-of-sample prediction. The paper does not report a held-out validation split, error bars, or repeated seeds for this choice. Separately, the written difficulty definition ('Easy if c≥1, Hard if c≤7') leaves no possible 'Medium' samples, an internal inconsistency that affects reproducibility but is not itself a circularity. Apart from this selection-on-evaluation issue, I see no self-citation, ansatz-smuggling, or definitional reduction, so the circularity score is a moderate 4 rather than higher.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the choice of data selection thresholds and the assumption that output variance indicates sample informativeness. No new physical or architectural entities are introduced.

free parameters (5)
  • tau_easy = 1
    Threshold for discrete data selection; samples with correct count <=1 are labeled easy. Chosen from ablation on benchmarks.
  • tau_hard = 7
    Threshold for discrete data selection; samples with correct count >=7 are labeled hard. Chosen from ablation.
  • delta_IoU_threshold = 0.3
    Minimum delta IoU for continuous data selection. Chosen from ablation in Table 4b.
  • delta_IoU_threshold_grounded = 0.1
    Relaxed threshold for grounded QA due to limited data. Chosen by design.
  • sample_counts = 16k, 8k, 8k
    Number of samples selected from each category. Chosen to balance data sources and performance.
assumptions (3)
  • domain assumption GRPO is an effective optimization algorithm for improving video-language reasoning
    The paper builds on DeepSeek-R1 and Video-R1 without independently verifying the choice.
  • domain assumption Medium-difficulty samples, as measured by output variance, are the most informative for GRPO tuning
    The paper supports this with ablations, but the proxy itself is assumed.
  • domain assumption The results on QwenVL-2.5-7B generalize to other VideoLLMs
    All experiments use a single base model; transferability is not tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency." pith.science (2026). https://pith.science/paper/CVGHA6G4

@misc{pith2026250601908,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CVGHA6G4}},
  note         = {Machine review of arXiv:2506.01908}
}
read the original abstract

Understanding real-world videos with complex semantics and long temporal dependencies remains a fundamental challenge in computer vision. Recent progress in multimodal large language models (MLLMs) has demonstrated strong capabilities in vision-language tasks, while reinforcement learning tuning (RLT) has further improved their reasoning abilities. In this work, we explore RLT as a post-training strategy to enhance the video-specific reasoning capabilities of MLLMs. Built upon the Group Relative Policy Optimization (GRPO) framework, we propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals. To facilitate effective preference-based optimization, we introduce a variance-aware data selection strategy based on repeated inference to identify samples that provide informative learning signals. We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA. Our method consistently outperforms supervised fine-tuning and existing RLT baselines, achieving superior performance with significantly less training data. These results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs. Notably, The initial code release (two months ago) has now been expanded with updates, including optimized reward mechanisms and additional datasets. The latest version is available at https://github.com/appletea233/Temporal-R1 .

Figures

Figures reproduced from arXiv: 2506.01908 by the authors.

Figure 1
Figure 1. Overview of our reward-based framework for post-training on video understanding tasks. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our training framework. Given a video and its corresponding instruction, the VideoLLM generates either an answer choice for multi-choice QA or a temporal segment for grounding. The model is optimized using a GRPO-based RLT pipeline with task-specific rewards, including a discrete reward for QA and a continuous temporal IoU reward for grounding. Additionally, we conducted extensive ablation studies to ver… view at source ↗
Figure 3
Figure 3. Data statistic of Temporal-RLT-Full-490k training dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Changes in data distribution before and after RLT. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Visualization of Temporal-RLT’s reasoning outputs on the Temporal Video Grounding task. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Visualization examples of Temporal-RLT’s reasoning output on Video QA task. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Visualization of Temporal-RLT’s reasoning output on Grounded QA task. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AdaThinkV: Adaptive Thinking for Token-Efficient Video Reasoning

    cs.CV 2026-08 conditional novelty 7.0 of 10

    A video reasoning model learns per question whether to reason aloud or answer directly, improving accuracy by about 3 points over the best adaptive baseline while using about 23% fewer output tokens.

  2. TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

    cs.CV 2026-07 conditional novelty 7.0 of 10

    TimeLens2 shows that a compact video MLLM can localize multiple evidence intervals in long videos by training on verified interval labels and a Wasserstein-based time-distance reward.

  3. TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Grounding

    cs.CV 2025-08 conditional novelty 7.0 of 10

    A reinforcement-learning method that forces video grounding models to emit progressively more accurate intermediate timestamps, improving accuracy and reasoning faithfulness without large teacher models.

  4. Empowering Nanoscale Connectivity through Molecular Communication: A Case Study of Virus Infection

    cs.NI 2025-08 unverdicted novelty 4.0 of 10

    A position paper proposing molecular communication as the link layer for epidemic-control bio-nano networks, with an ORF3a-based mutation identification simulation; the provided manuscript body does not match this abstract.

Reference graph

Works this paper leans on

41 extracted references · 2 canonical work pages · cited by 4 Pith papers

  1. [1]

    Localizing moments in video with natural language

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. InProceedings of the IEEE international conference on computer vision, pages 5803–5812, 2017

    2017

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  3. [3]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    arXiv 2024

  4. [4]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    arXiv 2025

  5. [5]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    arXiv 2024

  6. [6]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. InProceedings of the IEEE international conference on computer vision, pages 5267–5275, 2017

    2017

  7. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  8. [8]

    Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding

    Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xiaoying Tang, Dianbo Sui, Qingbin Liu, Xi Chen, and Kevin Zhao. Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 3302–3310, 2025. 10

    work page 2025

Show all 41 references
  1. [9]

    Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection.arXiv preprint arXiv:2411.14794, 2024

    Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection.arXiv preprint arXiv:2411.14794, 2024

    2024 arXiv

  2. [10]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14271–14280, 2024

    2024

  3. [11]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. InEuropean Conference on Computer Vision, pages 202–218. Springer, 2024

    2024

  4. [12]

    Unleashing the temporal-spatial reasoning capacity of gpt for training-free audio and language referenced video object segmentation

    Shaofei Huang, Rui Ling, Hongyu Li, Tianrui Hui, Zongheng Tang, Xiaoming Wei, Jizhong Han, and Si Liu. Unleashing the temporal-spatial reasoning capacity of gpt for training-free audio and language referenced video object segmentation. InProceedings of the AAAI Conference on A...

    2025

  5. [13]

    Dense- captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. InProceedings of the IEEE international conference on computer vision, pages 706–715, 2017

    2017

  6. [14]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  7. [15]

    Llava-st: A multimodal large language model for fine-grained spatial-temporal understanding.arXiv preprint arXiv:2501.08282, 2025

    Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao, Xiaoming Wei, and Si Liu. Llava-st: A multimodal large language model for fine-grained spatial-temporal understanding.arXiv preprint arXiv:2501.08282, 2025

    2025 arXiv

  8. [16]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  9. [17]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    2024

  10. [18]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. InEuropean Conference on Computer Vision, pages 323–340. Springer, 2024

    2024

  11. [19]

    Kangaroo: A powerful video-language model supporting long-context video input.arXiv preprint arXiv:2408.15542, 2024

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input.arXiv preprint arXiv:2408.15542, 2024

    2024 arXiv

  12. [20]

    St-llm: Large language models are effective temporal learners

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. InEuropean Conference on Computer Vision, pages 1–18. Springer, 2024

    2024

  13. [21]

    Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    2024 arXiv

  14. [22]

    Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  15. [23]

    Perception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Systems, 36:42748–42761, 2023

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Syste...

    2023

  16. [24]

    Momentor: Advancing video large language model with fine-grained temporal reasoning.arXiv preprint arXiv:2402.11435, 2024

    Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting Zhuang, and Siliang Tang. Momentor: Advancing video large language model with fine-grained temporal reasoning.arXiv preprint arXiv:2402.11435, 2024

    2024 arXiv

  17. [25]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14313–14323, 2024

    2024

  18. [26]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  19. [27]

    Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  20. [28]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  21. [29]

    Grounded-videollm: Sharpening fine-grained temporal grounding in video large language models.arXiv preprint arXiv:2410.03290, 2024

    Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videollm: Sharpening fine-grained temporal grounding in video large language models.arXiv preprint arXiv:2410.03290, 2024

    arXiv

  22. [30]

    Internvideo2

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2. 5: Empowering video mllms with long and rich context modeling.arXiv preprint arXiv:2501.12386, 2025

    2025 arXiv

  23. [31]

    Hawkeye: Training video-text llms for grounding text in videos.arXiv preprint arXiv:2403.10228, 2024

    Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan Zhao. Hawkeye: Training video-text llms for grounding text in videos.arXiv preprint arXiv:2403.10228, 2024

    2024 arXiv

  24. [32]

    Next-qa: Next phase of question- answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021

    2021

  25. [33]

    Can i trust your answer? visually grounded video question answering

    Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13204–13214, 2024

    2024

  26. [34]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models.arXiv preprint arXiv:2408.04840, 2024

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models.arXiv preprint arXiv:2408.04840, 2024

    2024 arXiv

  27. [35]

    Unhackable temporal rewarding for scalable video mllms.arXiv preprint arXiv:2502.12081, 2025

    En Yu, Kangheng Lin, Liang Zhao, Yana Wei, Zining Zhu, Haoran Wei, Jianjian Sun, Zheng Ge, Xiangyu Zhang, Jingyu Wang, et al. Unhackable temporal rewarding for scalable video mllms.arXiv preprint arXiv:2502.12081, 2025

    2025 arXiv

  28. [36]

    Timesuite: Improving mllms for long video understanding via grounded tuning.arXiv preprint arXiv:2410.19702, 2024

    Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, et al. Timesuite: Improving mllms for long video understanding via grounded tuning.arXiv preprint arXiv:2410.19702, 2024

    arXiv

  29. [37]

    Vision-r1: Evolving human-free alignment in large vision-language models via vision- guided reinforcement learning.arXiv preprint arXiv:2503.18013, 2025

    Yufei Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang. Vision-r1: Evolving human-free alignment in large vision-language models via vision- guided reinforcement learning.arXiv preprint arXiv:2503.18013, 2025

    2025 arXiv

  30. [38]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    2025 arXiv

  31. [39]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 12

    2024 arXiv

  32. [40]

    Mmvu: Measuring expert-level multi-discipline video understanding.arXiv preprint arXiv:2501.12380, 2025

    Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, et al. Mmvu: Measuring expert-level multi-discipline video understanding.arXiv preprint arXiv:2501.12380, 2025

    2025 arXiv

  33. [41]

    aha moment

    Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1-zero’s" aha moment" in visual reasoning on a 2b non-sft model.arXiv preprint arXiv:2503.05132, 2025. 13 Table 6: Result for Treating VideoQA as Grounded QA. Infer as GQA General VideoQA Re...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.