Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RL post-training lifts video caption accuracy with only 1.5k samples

desk verdict A credible, data-efficient RL recipe for video captioning, but the RL-vs-SFT claim is clouded because the reward and the benchmarks share the same Qwen2.5-72B judge protocol. read the letter →

arxiv 2506.01725 v1 pith:4YZ3EUAY submitted 2025-06-02 cs.CV

Desen Meng , Rui Huang , Zhilin Dai , Xinhao Li , Yifan Xu , Jun Zhang , Zhenpeng Huang , Meng Zhang
show 3 more authors
Lingshu Zhang Yi Liu Limin Wang
This is my paper · ORCID
classification cs.CV
keywords videocaptioningreinforcementlearningGRPOmultimodallargelanguagemodelstructuredthinkingrewardmodelingactiondescriptionchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that group-relative policy optimization (GRPO) reinforcement learning can improve open-ended video captioning in a 7B multimodal language model, using only 1.5k training videos and without large supervised datasets. The proposed model, VideoCap-R1, is prompted to first produce structured thinking that lists each video subject's attributes and actions, then write a full caption; two rewards score the thinking and the caption. On three benchmarks the authors report consistent gains over the Qwen2-VL-7B baseline (DREAM-1K +4.4 event F1, VDC +4.2 accuracy, CAREBENCH +3.1 action F1 and +6.9 object F1). The central comparison is against SFT on the same data, where RL wins even though SFT with structured thinking fails to help, which the authors attribute to teacher-forcing imitating format rather than learning the reasoning-to-caption link. If correct, this matters because it suggests reasoning-style RL can reduce the cost of improving captioning quality in video MLLMs.

What carries the argument

The load-bearing mechanism is a two-stage generation policy trained with GRPO under a composite reward: a format score, a think score, and a caption score. The think score (Tscore = 0.6×action F1 + 0.4×attribute F1) is obtained by solving bipartite matchings between predicted and annotated subjects, actions, and attributes, with similarity measured by the dot product of M3-Embedding vectors; the ground-truth annotation is itself generated by Qwen2.5-72B from source captions. The caption score is an event-coverage score: Qwen2.5-72B decides which ground-truth events are entailed by the generated caption, and the reward is 0, 0.5, or 1 depending on how much of the event list is covered. The composite reward makes the optimization signal decomposable: the model must learn to enumerate the right subjects and actions before it can collect the caption reward.

What would settle it

Take 100 held-out videos and have human annotators rate VideoCap-R1 captions against SFT and baseline captions for factual correctness; if human preference does not align with the reported F1/accuracy gains, the rewards are optimizing a proxy. A sharper version: retrain with human-written structured annotations instead of Qwen2.5-72B-generated ones and check whether the DREAM-1K event-F1 gain of +4.4 persists.

Watch

Extended reading notes

Core claim

VideoCap-R1 claims that the R1-style reinforcement-learning recipe transfers to open-ended video captioning when the task is decomposed into a structured thinking phase and a caption phase. The model first emits, inside a think tag, a subject-by-subject list of attributes and action sequences, then writes the final description in an answer tag. A think score, computed without an LLM by bipartite matching of action and attribute embeddings against structured annotations, rewards the reasoning content, while a caption score, computed by a Qwen2.5-72B judge through event-coverage entailment, rewards the final text. Training with GRPO on 1.5k curated videos improved event/action/object metrics on DREAM-1K, VDC, and CAREBENCH relative to both the base model and SFT-trained models; the authors further report that adding structured thinking to SFT degraded results, which they interpret as evidence that only RL establishes a genuine connection between reasoning and description. The caption-level gain on object F1 for CAREBENCH (34.3) surpasses GPT-4o-mini reported on the same benchmark.

Load-bearing premise

The reward signals assume the Qwen2.5-72B-generated structured annotations and event-entailment judgments are faithful ground truth, so if those targets are noisy or biased, the policy could learn to please the judge rather than describe videos accurately.

Editorial extensions

If this is right

  • On the authors' evidence, GRPO with dual think/caption rewards is more data-efficient than SFT for improving video captioning: 1.5k samples beat SFT on the same budget across DREAM-1K, VDC, and CAREBENCH.
  • Adding structured thinking to SFT can hurt performance, so the benefit of reasoning-style data depends on the training algorithm, not just the data format.
  • A think reward that scores intermediate content is sufficient to shift the model's caption behavior, even when the final caption is scored by a separate signal.
  • The caption reward's event-coverage formulation reduces, in the paper's experiments, the reward hacking seen with direct LLM completeness/naturalness scoring.
  • Scaling this pipeline to larger training sets is the stated next step, implying the 1.5k-sample result is treated as a lower bound of feasible gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported gains hold under blind human evaluation, the same think/caption reward decomposition could transfer to other open-ended generation tasks, such as dense video grounding or embodied instruction following, where final outputs cannot be matched to a single ground truth.
  • The reward targets are generated by an LLM judge; our read is that the practical ceiling of VideoCap-R1 may be set by the alignment between Qwen2.5-72B's structured annotations and entailment judgments and human annotation, so the method should be re-run with human-written structured labels to test how much of the gain is judge-dependent.
  • Because the 1.5k training videos come from the Tarsier2-Recap-585K pipeline and DREAM-1K is built in the same lineage, a decontamination check is needed before concluding the DREAM-1K gain is purely capability rather than distribution overlap; this is our caution, not the paper's claim.
  • A cheap testable extension is to ablate the think prompt at inference: if the model is asked to final-answer only, the gains should shrink but not vanish if the RL-trained policy internalized the structured reasoning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes VideoCap-R1, a GRPO-based reinforcement learning post-training method for video captioning. Starting from Qwen2-VL-7B, the model is prompted to first emit structured thinking that identifies subjects, attributes, and actions, and then produce a full caption. Two reward components are introduced: an LLM-free think score (Tscore) computed by embedding-based bipartite matching against structured annotations, and an LLM-assisted caption score (Escore) based on Qwen2.5-72B event-entailment judgments. Training on a curated 1.5k-video subset of Tarsier2-Recap-585K, the paper reports gains over the Qwen2-VL-7B baseline on DREAM-1K, VDC, and CAREBENCH, and shows that its GRPO-based approach outperforms SFT trained on the same data. Ablations examine the contribution of each reward component and compare Escore with a direct LLM scoring variant (CNscore).

Significance. If the empirical claims hold, this is a useful contribution: it is one of the first systematic applications of GRPO to open-ended video captioning, and the central comparison in Table 2—RL versus SFT on identical 1.5k training samples—is an internally consistent and relevant experiment. The paper also provides a careful decomposition of the reward into thinking and caption components, with ablations for each, and shows qualitative examples of finer-grained action description. The main risk is that the reward signal and the evaluation metrics are produced by the same Qwen2.5-72B judge, so the reported gains may partly reflect overfitting to that judge rather than true caption-quality improvement. The lack of multiple seeds and the absence of a decontamination check for the Tarsier-derived training set further weaken the current evidence. With additional validation, the method could be a valuable data-efficient alternative to SFT for video captioning.

major comments (4)
  1. [Sec. 3.2.3, Eq. (8); Sec. 4.1] The Escore reward is computed by Qwen2.5-72B entailment judgments using the same prompt as Tarsier, and the DREAM-1K event F1 metric is defined by the same Tarsier protocol; VDCSCORE and CAREBENCH action/object F1 also rely on model-based similarity. The RL policy is therefore trained to optimize a reward that is essentially the evaluation metric. The paper does not provide cross-judge validation (e.g., GPT-4o or Llama), human ratings, or a correlation analysis between Escore and human judgments. Without such evidence, the claimed gains of +4.4/+4.2/+3.1/+6.9 and the conclusion that GRPO is superior to SFT remain ambiguous, because both could reflect overfitting to the Qwen2.5-72B judge.
  2. [Table 2; Sec. 4.2.1] All reported results come from a single run, with no error bars or multiple seeds. The differences between SFT and RL, such as 32.8 vs. 34.2 DREAM-1K event F1 and 31.5 vs. 34.4 CAREBENCH action F1, are modest and could be within run-to-run variance. To support the central claim of GRPO superiority, the authors should report the mean and variance over at least three independent training runs, or otherwise justify that the observed differences are statistically meaningful.
  3. [Appendix B; Sec. 4.1] The training set is sampled from Tarsier2-Recap-585K, and the DREAM-1K benchmark originates from the Tarsier/LLM-based evaluation family. The paper does not report any decontamination or overlap analysis between the 1.5k training videos and the DREAM-1K, VDC, or CAREBENCH test sets. If overlap exists, the DREAM-1K improvements could be inflated. The authors should measure exact or near-duplicate video overlap and report results on the non-overlapping subset.
  4. [Eq. (6), Eq. (8), Appendix A] The thresholds δ1=0.28 and δ2=0.35 in Escore and the weights 0.6/0.4 in Tscore are fixed values without sensitivity analysis. If these hyperparameters were selected using the evaluation benchmarks, the held-out claim is weakened. The authors should provide a sensitivity study over δ1, δ2, and the Tscore weights, or otherwise justify that the results are robust to reasonable variations.
minor comments (5)
  1. [Table 3] The fourth data row, reporting values 32.5, 46.8, 35.2, 31.6, and 36.5, is missing its model label; from the text it appears to be the CNscore configuration, but the table should make this explicit.
  2. [Table 2] The last column header appears as 'A VG' and should be 'AVG' (average).
  3. [Figure 2] The three subfigures lack axis labels and legends, making it difficult to know which curve corresponds to which metric and what the x-axis represents.
  4. [Sec. 3.2.2] The think scorer is described as LLM-free, which is accurate at scoring time, but the ground-truth structured annotations used for matching are generated by Qwen2.5-72B from source captions (Appendix B). The paper should clarify this distinction to avoid implying that the annotations are human-verified.
  5. [Sec. 3.2.3] The statement that the Escore prompt is 'the same as Tarsier' should cite the exact prompt or include it in the appendix, since the equivalence between the reward prompt and the evaluation protocol is load-bearing for the central claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed gains are empirical benchmark results, and the reward/evaluation judge overlap is a validity risk, not a derivation-level circularity.

full rationale

VideoCap-R1 makes an empirical, not a derivation-level, claim: GRPO-trained VideoCap-R1 outperforms Qwen2-VL-7B and SFT variants on DREAM-1K, VDC, and CAREBENCH. The training rewards (Tscore and Escore) are defined from reference captions and a fixed Qwen2.5-72B judge; they are not fitted to the test benchmarks, and no test-set number enters the training objective. The think score compares predicted structured annotations to annotations generated from source captions; this is a standard supervised reward target, not a self-definitional prediction. The caption score Escore asks Qwen2.5-72B to check event entailment, and DREAM-1K's event-F1 metric uses a similar LLM-based protocol; this overlap is a legitimate external-validity and reward-overoptimization concern, but it does not make the reported gains equal to the training inputs by construction. SFT baselines were not trained against that judge, and the RL policy could fail to optimize it, so GRPO's relative advantage remains an empirical finding. No load-bearing self-citation, imported uniqueness theorem, or fitted parameter renamed as a prediction was found. Citations to VideoChat-R1 and Tarsier are contextual or benchmark references, not circular justifications. Accordingly, no specific circular step can be exhibited.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method contributes a reward design, not a derivation. It relies on four free parameters (weights and thresholds) and on three domain assumptions about the fidelity of LLM-generated annotations, embedding similarity, and judge-based event entailment, plus an unstated assumption of no train/eval contamination. No new entities are invented.

free parameters (4)
  • Tscore action/attribute weights = 0.6 and 0.4
    Eq. 6 sets Tscore = 0.6 x action F1 + 0.4 x attribute F1 with no sensitivity analysis; the balance is hand-chosen and directly shapes the reward.
  • Event coverage thresholds delta1, delta2 = 0.28 and 0.35
    Eq. 8 and Appendix A; no ablation or statement that these were fixed before test evaluation.
  • Think-score similarity threshold delta = not reported
    Sec. 3.2.2 zeroes embedding-similarity edges below an unreported threshold before matching; the reward is underspecified without it.
  • Final reward combination weights = 1, 1, 1 (Format, Tscore, Escore)
    Sec. 3.3 sums three terms with equal implicit weights; no study of alternative balancing.
assumptions (5)
  • domain assumption Structured thinking annotations produced by Qwen2.5-72B from source captions accurately represent the true subjects, attributes, and actions in each video.
    Appendix B constructs the 1.5k training set's thinking labels by prompting Qwen2.5-72B with captions from Tarsier2-Recap-585K; the think-score reward then optimizes against these labels.
  • domain assumption Embedding dot-product similarity from M3-Embedding is a valid semantic similarity measure for matching predicted and ground-truth actions/attributes.
    Eqs. 2-5 use dot products of M3-Embedding vectors with a threshold to compute precision/recall/F1. If embedding similarity misranks synonyms or fine-grained actions, the think score is noisy.
  • domain assumption Qwen2.5-72B event-entailment judgments (with Tarsier prompt) accurately determine which ground-truth events are covered by a generated caption.
    Sec. 3.2.3 defines Escore via event coverage judged by Qwen2.5-72B. The final reward and, through it, the trained policy depend on this judge being a faithful proxy for caption quality.
  • standard math The GRPO clipped surrogate objective with KL regularization is a valid and stable RL update for this policy and reward.
    Sec. 3.1.1, Eq. 1. Standard RL method from prior literature; not in question.
  • domain assumption No contamination between training videos sampled from Tarsier2-Recap-585K and the evaluation benchmarks, especially DREAM-1K, which comes from the same Tarsier lineage.
    Appendix B uses Tarsier2-Recap-585K; DREAM-1K is from Tarsier [33]. The paper reports no explicit decontamination analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking." pith.science (2026). https://pith.science/paper/4YZ3EUAY

@misc{pith2026250601725,
  author       = {Pith},
  title        = {Pith review of: VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4YZ3EUAY}},
  note         = {Machine review of arXiv:2506.01725}
}
read the original abstract

While recent advances in reinforcement learning have significantly enhanced reasoning capabilities in large language models (LLMs), these techniques remain underexplored in multi-modal LLMs for video captioning. This paper presents the first systematic investigation of GRPO-based RL post-training for video MLLMs, with the goal of enhancing video MLLMs' capability of describing actions in videos. Specifically, we develop the VideoCap-R1, which is prompted to first perform structured thinking that analyzes video subjects with their attributes and actions before generating complete captions, supported by two specialized reward mechanisms: a LLM-free think scorer evaluating the structured thinking quality and a LLM-assisted caption scorer assessing the output quality. The RL training framework effectively establishes the connection between structured reasoning and comprehensive description generation, enabling the model to produce captions with more accurate actions. Our experiments demonstrate that VideoCap-R1 achieves substantial improvements over the Qwen2VL-7B baseline using limited samples (1.5k) across multiple video caption benchmarks (DREAM1K: +4.4 event F1, VDC: +4.2 Acc, CAREBENCH: +3.1 action F1, +6.9 object F1) while consistently outperforming the SFT-trained counterparts, confirming GRPO's superiority in enhancing MLLMs' captioning capabilities.

Figures

Figures reproduced from arXiv: 2506.01725 by the authors.

Figure 1
Figure 1. Motivation of VideoCap-R1. SFT requires costly high-quality data and the trained model merely imitates training distributions. VideoCap-R1 instead decomposes captioning into structured thinking and answering phases, optimized via GRPO with dual rewards for thinking and caption. By effectively establishing the connection between structured reasoning and comprehensive description generation, VideoCap-R1 can generate c… view at source ↗
Figure 2
Figure 2. The metric curves of think score, total reward and response length of [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison between VideoCap-R1 and baseline models. We annotate exam [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The Format of structured thinking process and final answer. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Prompt Template for Structured Thinking Generation. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: The first qualitative comparison between VideoCap-R1 and baseline models. The video [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The second qualitative comparison between VideoCap-R1 and baseline models. The [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: The third qualitative comparison between VideoCap-R1 and baseline models. The video [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The fourth qualitative comparison between VideoCap-R1 and baseline models. The [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The fifth qualitative comparison between VideoCap-R1 and baseline models. The video is [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Prompt Template for Completeness Score. E The Training Prompt for Qwen2-VL-7B used in GRPO [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Prompt Template for Naturalness Score. Describe the video in detail using the following process. Think about the details in the video: Firstly, think how many subjects there are in the video, what attributes each subject has, and then provide the actions/motions seque…
Figure 13
Figure 13. Figure 13: Prompt Template for Training. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Writing editing instructions that explicitly bind each attribute to a reference image via `<Image_N>` tokens substantially improves multi-reference video editing, and a specialized MLLM trained with GRPO generates the...

  2. PercepCap: Video Captioner with Structured Spatio-Temporal Perception

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Explicitly generating object trajectories and event timestamps before the final caption improves detailed video captioning on multiple benchmarks, though the improvement is largely driven by an external perception oracle.

  3. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Reference graph

Works this paper leans on

52 extracted references · 9 canonical work pages · cited by 3 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition , pages 961–970, 2015

    2015

  3. [3]

    Auroracap: Efficient, performant video detailed captioning and a new benchmark

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq-Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. In The Thirteenth International Conference on Learning Representations,

    work page

  4. [4]

    M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge dis- tillation

    Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge dis- tillation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Associa- tion for Computational Linguistics: ACL 2024 , pages 2318–2335, Bangko...

    doi:10.18653/v1/2024.findings-acl.137 2024

  5. [5]

    Sharegpt4video: Improving video understanding and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325, 2024

    arXiv 2024

  6. [6]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  7. [7]

    Video-r1: Reinforcing video reasoning in mllms

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776, 2025

    arXiv 2025

  8. [8]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    arXiv 2024

Show all 52 references
  1. [9]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision , pages 5267–5275, 2017

    2017

  2. [10]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning , pages 10835–10866. PMLR, 2023

    2023

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [12]

    Shot2story20k: A new benchmark for comprehensive understanding of multi-shot videos

    Mingfei Han, Linjie Yang, Xiaojun Chang, and Heng Wang. Shot2story20k: A new benchmark for comprehensive understanding of multi-shot videos. arXiv preprint arXiv:2312.10300, 2023

    2023 arXiv

  5. [13]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402....

    2024 arXiv

  6. [14]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826, 2025

    2025 arXiv

  7. [15]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  8. [16]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  9. [17]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. 10

    2024 arXiv

  10. [18]

    A shortest augmenting path algorithm for dense and sparse linear assignment problems

    Roy Jonker and Ton V olgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. In Helmut Schellhaas, Paul van Beek, Heinz Isermann, Reinhart Schmidt, and Mynt Zijlstra, editors, DGOR/NSOR, pages 622–622, Berlin, Heidelberg, 1988. Springer ...

    1988

  11. [19]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  12. [20]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22195–2...

    2024

  13. [21]

    Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958, 2025

    2025 arXiv

  14. [22]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023

    2023

  15. [23]

    Visual-rft: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  16. [24]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  17. [25]

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  18. [26]

    Introducing gemini 2.0: our new ai model for the agentic era, 2024

    Sundar Pichai, D Hassabis, and K Kavukcuoglu. Introducing gemini 2.0: our new ai model for the agentic era, 2024

    2024

  19. [27]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  20. [28]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  21. [29]

    Vlm-r1: A stable and generalizable r1-style large vision-language model

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  22. [30]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  23. [31]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  24. [32]

    Llamav-o1: Rethinking step-by-step visual reasoning in llms

    Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-o1: Rethinking step-by-step visual reasoning in llms. arXiv preprint arXiv:2501.06186, 2025

    2025 arXiv

  25. [33]

    Tarsier: Recipes for training and evaluating large video description models

    Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634, 2024

    2024 arXiv

  26. [34]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  27. [35]

    Timezero: Temporal video grounding with reasoning-guided lvlm

    Ye Wang, Boshen Xu, Zihao Yue, Zihan Xiao, Ziheng Wang, Liang Zhang, Dingyi Yang, Wenxuan Wang, and Qin Jin. Timezero: Temporal video grounding with reasoning-guided lvlm. arXiv preprint arXiv:2503.13377, 2025. 11

    2025 arXiv

  28. [36]

    Llava-o1: Let vision language models reason step-by-step

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440, 2024

    2024 arXiv

  29. [37]

    Fine-grained video-text retrieval: A new benchmark and method

    Yifan Xu, Xinhao Li, Yichun Yang, Rui Huang, and Limin Wang. Fine-grained video-text retrieval: A new benchmark and method. arXiv preprint arXiv:2501.00513, 2024

    2024 arXiv

  30. [38]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  31. [39]

    Vript: A video is worth thousands of words

    Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. Advances in Neural Information Processing Systems , 37:57240–57261, 2024

    2024

  32. [40]

    Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024

    2024 arXiv

  33. [41]

    Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319, 2024

    2024 arXiv

  34. [42]

    Modeling context in referring expressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 69–85. Springer, 2016

    2016

  35. [43]

    Tarsier2: Advancing large vision-language models from detailed video description to comprehensive video understanding

    Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advancing large vision-language models from detailed video description to comprehensive video understanding. arXiv preprint arXiv:2501.07888, 2025

    2025 arXiv

  36. [44]

    7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient

    Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https: //hkust-nlp.notion.site/simplerl-reason , 2025. Notion Blog

    2025

  37. [45]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Sprin...

    2024

  38. [46]

    Improve vision language model chain-of-thought reasoning

    Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. Improve vision language model chain-of-thought reasoning. arXiv preprint arXiv:2410.16198, 2024

    2024 arXiv

  39. [47]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  40. [48]

    R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning

    Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning. arXiv preprint arXiv:2503.05379, 2025

    2025 arXiv

  41. [49]

    Mmvu: Measuring expert-level multi-discipline video understanding

    Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, et al. Mmvu: Measuring expert-level multi-discipline video understanding. arXiv preprint arXiv:2501.12380, 2025

    2025 arXiv

  42. [50]

    Swift:a scalable lightweight infrastructure for fine-tuning

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scalable lightweight infrastructure for fine-tuning. CoRR, abs/2408.05517, 2024

    2024 arXiv

  43. [51]

    aha moment

    Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1-zero’s" aha moment" in visual reasoning on a 2b non-sft model. arXiv preprint arXiv:2503.05132, 2025. 12 Appendix This supplementary material includes the following sections: • In Section ...

    2025 arXiv

  44. [2025]

    URL https://openreview.net/forum?id=tTDUrseRRU

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.