Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Reinforcing Video Reasoning with Focused Thinking

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Video QA reinforcement learning can be made far cheaper: a 500-step, token-weighted GRPO run on 1,000 CLEVRER samples reaches 50.4% counterfactual accuracy versus 31.6% for Video-R1's 4K-step pipeline.

desk verdict The token-weighting and soft-reward components are worth a look, but the headline CLEVRER gain is an evaluation artifact, not a real model improvement. read the letter →

arxiv 2505.24718 v3 pith:CD3FNEPY submitted 2025-05-30 cs.CV

Jisheng Dang , Jingze Wu , Teng Wang , Xuanhui Lin , Nannan Zhu , Hongbo Chen , Wei-Shi Zheng , Meng Wang
show 1 more author
Tat-Seng Chua
This is my paper
classification cs.CV
keywords videoreasoningreinforcementlearningGRPOtoken-levelimportanceweightingmulti-answersoftrewardquestion-answerinversionmultimodallargelanguagemodelscounterfactualQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that two fixable inefficiencies dominate reinforcement-learning fine-tuning of video-language models: verbose, unfocused reasoning chains and all-or-nothing binary rewards. Its remedy, TW-GRPO (token-weighted group-relative policy optimization), reweights each token in the policy-gradient loss by its informational density, estimated as the cross-sample divergence of the token's predictive distribution, and replaces 0/1 rewards with partial credit $|P|/|G|$ on multi-answer questions that are generated by negating existing single-choice items. Trained for 500 RL steps on 1,000 CLEVRER counterfactual samples starting from Qwen2.5-VL-7B, the model reaches 50.4% accuracy on CLEVRER (18.8 points above Video-R1) and 65.8% on MMVU, while producing shorter reasoning chains. If these results are right, a small GPU budget is enough to push a video reasoner past pipelines that use far more data and compute.

What carries the argument

The load-bearing mechanism is the token-level importance weight $w_t$ inserted into a sequence-normalized GRPO objective. The weight is a min-max-normalized, $(1+\alpha)$-scaled Kullback-Leibler divergence of each position's predictive distribution from the group-averaged expected distribution, with variable-length outputs padded by the uniform distribution over the vocabulary so that short sequences still contribute; it exists to counteract a gradient-cancellation effect derived in Appendix B, where informative tokens receive opposing gradients across trajectories and their learning signal washes out. The second mechanism is the multi-answer soft reward $R_{\mathrm{soft}}$ of Equation 7, which converts a sparse 0/1 signal into graded credit proportional to the overlap of predicted and ground-truth answer sets. The third is question-answer inversion, which builds multi-answer training questions from single-choice data by negation and by randomly dropping correct options; the ablations in Tables 2 and 3 attribute the overall gains to all three components working together.

What would settle it

Take the released Video-R1 checkpoint and re-score it on the same indefinite-choice CLEVRER counterfactual test set, using the paper's comma-separated-letter prompt and its single- and multiple-choice accuracy rules; if Video-R1's accuracy climbs from 31.6% toward 50.4%, the headline margin is an artifact of the evaluation protocol. The mirror check is to score TW-GRPO under the original single-answer protocol, and if its accuracy falls back toward 31.6%, the same conclusion follows.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that GRPO for video reasoning can be upgraded by two coordinated changes. At token position $t$, the method computes a divergence score $D_t=\sum_{i=1}^{G}D_{\mathrm{KL}}(p(o_{i,t})\,\|\,\mathbb{E}[o_t])$ across the group of sampled responses, normalizes it to a weight $w_t=(1+\alpha)(D_t-D_{\min})/(D_{\max}-D_{\min})$, and multiplies $w_t$ into each token's clipped policy-gradient term, so positions where the model's sampled outputs disagree receive larger updates while filler tokens such as 'Let's think...' receive near-zero weight. On the reward side, single-choice QA is recast as multi-answer QA with $R_{\mathrm{soft}}=|P|/|G|$ when the predicted set $P$ is a subset of the ground-truth set $G$ and $0$ otherwise, and question-answer inversion manufactures multi-answer training samples from single-choice benchmarks by negating the question and randomly removing correct options. The paper reports 50.4% on CLEVRER counterfactual reasoning, 76.1% on NExT-GQA, and 65.8% on MMVU, plus competitive scores on MVBench, TempCompass, and VideoMME, together with faster convergence of reward variance and shorter generated completions during training.

Load-bearing premise

The load-bearing premise is that the CLEVRER comparison is apples-to-apples: the paper trains and tests under an indefinite-choice protocol that accepts comma-separated multiple answers, while Video-R1's 31.6% comes from the original single-answer counterfactual benchmark, so if the protocols differ, the announced 18.8-point gain is largely a scoring difference rather than a model improvement.

Editorial extensions

If this is right

  • RL fine-tuning of video reasoners is far cheaper than prior pipelines suggested: 500 steps on 1,000 CLEVRER counterfactual samples beat a pipeline that used 165K SFT samples plus 4K RL steps on CLEVRER, NExT-GQA, and MMVU.
  • Multi-answer soft rewards lower reward variance during training, making policy updates stable enough for harder, partially correct answers.
  • Token weighting produces shorter completions and qualitatively more focused reasoning chains that extract the relevant causal and quantitative cues from video.
  • The recipe transfers beyond counterfactual reasoning: MVBench, TempCompass, and VideoMME scores match or exceed earlier RL-tuned baselines, so the gains are not confined to CLEVRER.
  • Single-choice benchmarks such as NExT-GQA and STAR can be recycled into multi-answer training data by question-answer inversion, removing a data-scarcity bottleneck for soft-reward RL.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline 18.8-point CLEVRER margin depends on the evaluation protocol, which the paper changes to an indefinite-choice format that accepts comma-separated multiple answers; re-scoring the Video-R1 checkpoint under that same protocol is needed before the gain can be attributed to the model rather than the metric.
  • Because token importance is computed from output distributions alone, the weight sequence is a ready-made interpretability probe: one could test whether high-weight positions coincide with human-annotated salient events in CLEVRER, a check the paper does not run.
  • The subset rule gives zero credit as soon as the prediction includes any false positive; a symmetric Jaccard-style credit variant would show whether the improvements come from graded credit in general or from this particular penalty scheme.
  • Nothing in the mechanism is video-specific, so the same token weighting plus multi-answer soft reward should transfer to any task with a verifiable answer set, such as multi-label document QA or multi-answer math problems, without architectural changes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TW-GRPO, an RL fine-tuning method for video MLLMs that extends GRPO in two directions: token-level importance weighting computed from within-group KL divergence, and multi-answer QA with a soft reward that gives partial credit for correct subsets. It also introduces question-answer inversion (QAI) to convert single-choice benchmarks into multi-answer training data. The authors report state-of-the-art results on CLEVRER (50.4% vs. Video-R1's 31.6%), NExT-GQA, and MMVU, using a lightweight 1K-step/1K-sample training budget, and provide ablations, sensitivity analyses, training dynamics, and qualitative reasoning-path comparisons.

Significance. If the empirical claims were measured under comparable protocols, the paper would make a useful efficiency-oriented contribution to RL fine-tuning for video reasoning: it addresses a real limitation of binary rewards and proposes a lightweight token-level importance signal without an auxiliary critic. The authors also release code and report training dynamics and ablations in unusual detail, which is valuable for reproducibility. However, the central empirical claim is currently undermined by a non-comparable evaluation protocol on CLEVRER, and the final training objective in Eq. (29) is not actually derived from the reformulated GRPO objective in Appendix B.2.1. These issues concern the headline results and the paper's claimed mechanism, so the contribution as written is not yet established.

major comments (4)
  1. [Appendix C, Table 2, Section 4.2] The CLEVRER evaluation is not comparable to the Video-R1 baseline. Appendix C states that CLEVRER uses a prompting strategy 'designed for our indefinite-choice setting,' requiring comma-separated answers when multiple answers exist, and Table 2 reports separate single-choice, multiple-choice, and all accuracy numbers. The paper's own GRPO baseline under this modified indefinite-choice protocol already reaches 41.1% on CLEVRER, 9.5 points above the published Video-R1 number of 31.6%, even before TW-GRPO's components are added. The claimed 18.8% improvement therefore largely reflects the modified evaluation protocol rather than a model gain. The authors need to evaluate TW-GRPO under the original single-answer CLEVRER counterfactual protocol and report that number; as written, the state-of-the-art claim is unsupported.
  2. [Appendix B.2.1-B.2.3, Eq. (29)] The final objective in Eq. (29) does not follow from the derivation in Eqs. (11)-(26). That derivation yields an objective in which the update at each token is scaled by 1/pi^emp_{theta_old,t} and the difference (pi_theta - pi^emp_{theta,t}), with no separate token-level weight w_t. In Eq. (29), w_t is inserted only into the first branch of the min, while the clip branch remains unweighted, which is not a consequence of the preceding derivation. The paper should either derive Eq. (29) rigorously or explicitly present the token weighting as a heuristic modification of Eq. (11) with theoretical motivation. As written, the mechanism claim is not adequately supported.
  3. [Section 4.3, Appendix D.1-D.2] Hyperparameters appear to be tuned on the same evaluation benchmarks used for the main results. Table A1 and Figure A1 report accuracy on CLEVRER and MMVU for different sampling numbers, temperatures, and alpha values, and the default alpha=0.7 is selected from these curves; no held-out validation split is described. Because the ablations in Table 3 are also computed on the same test sets, the reported gains may partly reflect selection on the evaluation distribution. The authors should either use a validation split for hyperparameter selection or disclose this selection explicitly. This is particularly relevant because the non-CLEVRER SOTA margins are small (1.6-1.8%).
  4. [Section 3.3, Eq. (7), Section 4.3] The soft reward and the 'soft accuracy' metric are the same formula (Eq. (7)), so the soft-accuracy improvements reported in Tables 2 and 3 are aligned by construction with the training objective and do not provide independent evidence of better reasoning. In addition, the reward assigns zero to any prediction containing a false positive, which is a design choice, but the paper should discuss the implications of this asymmetry when claiming that the reward distinguishes partial correctness.
minor comments (5)
  1. [Section 3.2] The section title 'Token-Level Importance Wighting' contains a typo; it should be 'Weighting'. Similarly, 'we purpose' should be 'we propose', and the abstract's 'we employs' should be 'we employ'.
  2. [Eq. (6)] The expectation in Eq. (6) is over {alpha_i}_{i=1}^G sampled from pi_theta_old, but alpha is introduced later as the scaling hyperparameter; this should be {o_i}.
  3. [Table 3] In the 'Effect of Token Weighting (multi-choice, Fixed Reward)' block, TW-GRPO with fixed reward reduces multiple-choice accuracy relative to GRPO (26.0% vs. 32.3%) while increasing single-choice accuracy (64.9% vs. 50.4%). The text states that token weighting shows notable gains in both fixed and soft reward settings, which overstates the fixed-reward result; this should be acknowledged or clarified.
  4. [Appendix D.2.2, Figure A2] The 'All Positions' configuration applies importance weights to padding positions, and Figure A2 shows that padding tokens receive relatively high weights. The claim that padding positions carry 'alignment or reasoning-related information' is not supported by any direct evidence; padding tokens are filled with a uniform distribution, so upweighting them may be an artifact of the divergence computation. This deserves a more cautious interpretation.
  5. [Figure A1] The caption states 'VideoR1 Avg: 66.2' and 'VideoChatR1 Avg: 65.7' but the figure does not define the 'Average' metric and the text does not explain how these numbers relate to the reported benchmark accuracies. Please clarify.

Circularity Check

1 steps flagged · score 2.0 of 10

The TW-GRPO derivation is self-contained; the only exhibitable circularity is auxiliary: 'soft accuracy' in Section 4.3 is defined by the paper's own reward formula (Eq. 7), so soft-reward improvements on that metric are partly tautological. The headline CLEVRER/MMVU gains are measured benchmark numbers, not quantities forced by the method's equations.

  1. self definitional [Section 3.3, Eq. (7); Section 4.3 Ablation Study]
    "Rsoft = ( |P| / |G| , if P ⊆ G; 0, if P is not subset of G ) ... In addition to using accuracy as an evaluation metric, we compute soft accuracy based on Equation 7, enabling a more fine-grained assessment of the model's performance on multi-choice datasets."

    The evaluation metric 'soft accuracy' is literally the training reward Rsoft of Eq. 7. Therefore, the ablation claim that soft-reward training improves soft accuracy measures the model against its own optimized objective; the metric does not supply independent evidence because the reward and the metric are the same formula. This is limited circularity rather than a fully forced result: hard accuracy and external benchmark numbers are also reported, so the central SOTA claims do not reduce to this metric alone.

full rationale

The paper's actual derivation chain is empirical and not self-referential. Token importance weights wt are computed from intra-group KL divergence (Eqs. 4-5) and inserted into a GRPO/DAPO-style objective (Eq. 29); the objective is not derived from the weights by identity, and the weights themselves are not fitted to the reported accuracies. The multi-answer soft reward (Eq. 7) is an exogenous reward design, and the main Table 1 numbers are externally measured accuracies on fixed benchmarks rather than quantities generated by the method's equations. The paper does not lean on a self-citation chain or a uniqueness theorem to force its choices; references to prior entropy-based methods are contextual, not load-bearing. The CLEVRER evaluation mismatch with Video-R1 (Appendix C prompt and Table 2 split into single/multiple-choice accuracies) is a serious external-validity and comparability concern, but it is not circularity: the 50.4% figure is a measured result under a self-designed protocol, not a number forced by the paper's definitions. The only step that reduces by construction is the auxiliary 'soft accuracy' metric, which is defined by the same formula as the training reward; because the central hard-accuracy and benchmark claims stand independently, the overall circularity score remains low at 2.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities or mediators are introduced. The token weights are computed quantities derived from the model's own output distributions, and the soft reward is a scalar function of the predicted and ground-truth sets. The free parameters are hyperparameters tuned on the evaluation benchmarks plus an unspecified QAI randomization rate.

free parameters (4)
  • alpha (token weight offset) = 0.7
    Chosen as default after sensitivity analysis on the same evaluation benchmarks (Figure A1); not held out.
  • Sampling group size G = 8
    Table A1 shows G=8 best on CLEVRER/MMVU; performance degrades at 4 and 12, so the method's success depends on this tuned value.
  • Sampling temperature = 1.0
    Table A1 shows temperature 1.0 is best; the method depends on the diversity-stability balance.
  • QAI random answer-removal probability = unspecified
    The augmentation randomly removes correct options, but the probability is never stated, so the training-data construction has an untracked free parameter.
assumptions (4)
  • domain assumption Assumption B.1: number of sampled trajectories G is sufficiently large and the policy produces stable outputs across similar histories
    Used to approximate the importance-ratio denominator with an empirical distribution (Eq. 20-21); only empirically verified on the same benchmarks in Appendix D.1.
  • ad hoc to paper Padding positions filled with a uniform distribution represent meaningful token positions for divergence computation
    Needed to align variable-length sequences for D_t and to justify weighting all positions including padding (Table A2); padding carries no semantics.
  • ad hoc to paper Negating the question and inverting the answer set yields valid ground truth for multi-answer training
    QAI assumes the complement of the original answer is the correct set after negation; no validation of label correctness is provided.
  • standard math Standard GRPO/PPO clipping and group-wise advantage normalization
    Borrowed from DeepSeekMath/GRPO and used in Eq. 2 without modification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcing Video Reasoning with Focused Thinking." pith.science (2026). https://pith.science/paper/CD3FNEPY

@misc{pith2026250524718,
  author       = {Pith},
  title        = {Pith review of: Reinforcing Video Reasoning with Focused Thinking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CD3FNEPY}},
  note         = {Machine review of arXiv:2505.24718}
}
read the original abstract

Recent advancements in reinforcement learning, particularly through Group Relative Policy Optimization (GRPO), have significantly improved multimodal large language models for complex reasoning tasks. However, two critical limitations persist: 1) they often produce unfocused, verbose reasoning chains that obscure salient spatiotemporal cues and 2) binary rewarding fails to account for partially correct answers, resulting in high reward variance and inefficient learning. In this paper, we propose TW-GRPO, a novel framework that enhances visual reasoning with focused thinking and dense reward granularity. Specifically, we employs a token weighting mechanism that prioritizes tokens with high informational density (estimated by intra-group information entropy), suppressing redundant tokens like generic reasoning prefixes. Furthermore, we reformulate RL training by shifting from single-choice to multi-choice QA tasks, where soft rewards enable finer-grained gradient estimation by distinguishing partial correctness. Additionally, we propose question-answer inversion, a data augmentation strategy to generate diverse multi-choice samples from existing benchmarks. Experiments demonstrate state-of-the-art performance on several video reasoning and general understanding benchmarks. Notably, TW-GRPO achieves 50.4\% accuracy on CLEVRER (18.8\% improvement over Video-R1) and 65.8\% on MMVU. Our codes are available at \href{https://github.com/longmalongma/TW-GRPO}.

Figures

Figures reproduced from arXiv: 2505.24718 by the authors.

Figure 1
Figure 1. TW-GRPO integrates focused thinking and soft multi-level rewards for multi-choice QA. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the TW-GRPO framework. The diagram shows the key steps in a forward [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Training dynamics of different GRPO variants. (a) TW-GRPO achieves faster con￾vergence in reward standard deviation, indicating more stable and efficient learning. (b) It also produces consistently shorter output lengths, reflecting more concise and effective reasoning than other methods. TW-GRPO demonstrates competitive performance, even with fewer training resources. In MVBench, TW-GRPO matches the zero-shot perfo… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of reasoning paths from T-GRPO and TW-GRPO on MMVU samples. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answering

    cs.CV 2025-06

Reference graph

Works this paper leans on

41 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Biet al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  2. [2]

    R1-v: Reinforcing super generalization ability in vision-language models with less than $3,

    L. Chen, L. Li, H. Zhao, Y . Song, and Vinci, “R1-v: Reinforcing super generalization ability in vision-language models with less than $3,” https://github.com/Deep-Agent/R1-V, 2025, accessed: 2025-02-02

    work page 2025

  3. [3]

    Vision- r1: Incentivizing reasoning capability in multimodal large language models,

    W. Huang, B. Jia, Z. Zhai, S. Cao, Z. Ye, F. Zhao, Z. Xu, Y . Hu, and S. Lin, “Vision- r1: Incentivizing reasoning capability in multimodal large language models,”arXiv preprint arXiv:2503.06749, 2025

    arXiv 2025

  4. [4]

    Mm- eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning,

    F. Meng, L. Du, Z. Liu, Z. Zhou, Q. Lu, D. Fu, T. Han, B. Shi, W. Wang, J. Heet al., “Mm- eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning,” arXiv preprint arXiv:2503.07365, 2025

    arXiv 2025

  5. [5]

    R1-zero’s

    H. Zhou, X. Li, R. Wang, M. Cheng, T. Zhou, and C.-J. Hsieh, “R1-zero’s" aha moment" in visual reasoning on a 2b non-sft model,”arXiv preprint arXiv:2503.05132, 2025

    arXiv 2025

  6. [6]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl,

    Y . Peng, G. Zhang, M. Zhang, Z. You, J. Liu, Q. Zhu, K. Yang, X. Xu, X. Geng, and X. Yang, “Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl,” arXiv preprint arXiv:2503.07536, 2025

    arXiv 2025

  7. [7]

    Video-r1: Reinforc- ing video reasoning in mllms,

    K. Feng, K. Gong, B. Li, Z. Guo, Y . Wang, T. Peng, B. Wang, and X. Yue, “Video-r1: Reinforc- ing video reasoning in mllms,”arXiv preprint arXiv:2503.21776, 2025

    arXiv 2025

  8. [8]

    Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning,

    X. Li, Z. Yan, D. Meng, L. Dong, X. Zeng, Y . He, Y . Wang, Y . Qiao, Y . Wang, and L. Wang, “Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning,”arXiv preprint arXiv:2504.06958, 2025

    arXiv 2025

Show all 41 references
  1. [9]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  2. [10]

    Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency,

    D. Jiang, R. Zhang, Z. Guo, Y . Li, Y . Qi, X. Chen, L. Wang, J. Jin, C. Guo, S. Yanet al., “Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency,”arXiv preprint arXiv:2502.09621, 2025

    2025 arXiv

  3. [11]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization,

    J. Zhang, J. Huang, H. Yao, S. Liu, X. Zhang, S. Lu, and D. Tao, “R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization,”arXiv preprint arXiv:2503.12937, 2025

    2025 arXiv

  4. [12]

    Clevrer: Collision events for video representation and reasoning,

    K. Yi, C. Gan, Y . Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum, “Clevrer: Collision events for video representation and reasoning,”arXiv preprint arXiv:1910.01442, 2019

    1910 arXiv

  5. [13]

    Can i trust your answer? visually grounded video ques- tion answering,

    J. Xiao, A. Yao, Y . Li, and T. S. Chua, “Can i trust your answer? visually grounded video ques- tion answering,” in2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  6. [14]

    Mmvu: Measuring expert-level multi-discipline video understanding,

    Y . Zhao, L. Xie, H. Zhang, G. Gan, Y . Long, Z. Hu, T. Hu, W. Chen, C. Li, J. Songet al., “Mmvu: Measuring expert-level multi-discipline video understanding,”arXiv preprint arXiv:2501.12380, 2025

    2025 arXiv

  7. [15]

    Openai o1 system card,

    A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carneyet al., “Openai o1 system card,”arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  8. [16]

    Secrets of rlhf in large language models part i: Ppo,

    R. Zheng, S. Dou, S. Gao, Y . Hua, W. Shen, B. Wang, Y . Liu, S. Jin, Q. Liu, Y . Zhouet al., “Secrets of rlhf in large language models part i: Ppo,”arXiv preprint arXiv:2307.04964, 2023

    2023 arXiv

  9. [17]

    Vision-r1: Evolving human-free alignment in large vision-language models via vision-guided reinforcement learning,

    Y . Zhan, Y . Zhu, S. Zheng, H. Zhao, F. Yang, M. Tang, and J. Wang, “Vision-r1: Evolving human-free alignment in large vision-language models via vision-guided reinforcement learning,” arXiv preprint arXiv:2503.18013, 2025. 10

    2025 arXiv

  10. [18]

    Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization,

    M. Chen, G. Chen, W. Wang, and Y . Yang, “Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization,”arXiv preprint arXiv:2505.12346, 2025

    2025 arXiv

  11. [19]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms,

    Z. Cheng, S. Leng, H. Zhang, Y . Xin, X. Li, G. Chen, Y . Zhu, W. Zhang, Z. Luo, D. Zhaoet al., “Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms,” arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  12. [20]

    Audio-visual llm for video understanding,

    F. Shu, L. Zhang, H. Jiang, and C. Xie, “Audio-visual llm for video understanding,”arXiv preprint arXiv:2312.06720, 2023

    2023 arXiv

  13. [21]

    Visa: Reasoning video object segmentation via large language models,

    C. Yan, H. Wang, S. Yan, X. Jiang, Y . Hu, G. Kang, W. Xie, and E. Gavves, “Visa: Reasoning video object segmentation via large language models,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 98–115

    2024

  14. [22]

    Towards self-improving systematic cognition for next-generation foundation mllms,

    X. Zhang, D. Peng, Y . Zhang, Z. Guo, C. Wu, C. Chen, W. Ke, H. Meng, and M. Sun, “Towards self-improving systematic cognition for next-generation foundation mllms,”arXiv preprint arXiv:2503.12303, 2025

    2025 arXiv

  15. [23]

    Llava-uhd v2: an mllm integrating high-resolution feature pyramid via hierarchical window transformer,

    Y . Zhang, Y . Liu, Z. Guo, Y . Zhang, X. Yang, C. Chen, J. Song, B. Zheng, Y . Yao, Z. Liuet al., “Llava-uhd v2: an mllm integrating high-resolution feature pyramid via hierarchical window transformer,”arXiv preprint arXiv:2412.13871, 2024

    2024 arXiv

  16. [24]

    Forking paths in neural text generation,

    E. Bigelow, A. Holtzman, H. Tanaka, and T. Ullman, “Forking paths in neural text generation,” arXiv preprint arXiv:2412.07961, 2024

    2024 arXiv

  17. [25]

    Critical tokens matter: Token-level contrastive estimation enhence llm’s reasoning capability,

    Z. Lin, T. Liang, J. Xu, X. Wang, R. Luo, C. Shi, S. Li, Y . Yang, and Z. Tu, “Critical tokens matter: Token-level contrastive estimation enhence llm’s reasoning capability,”arXiv preprint arXiv:2411.19943, 2024

    2024 arXiv

  18. [26]

    Dapo: An open-source llm reinforcement learning system at scale,

    Q. Yu, Z. Zhang, R. Zhu, Y . Yuan, X. Zuo, Y . Yue, T. Fan, G. Liu, L. Liu, X. Liuet al., “Dapo: An open-source llm reinforcement learning system at scale,”arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  19. [27]

    Mvbench: A comprehensive multi-modal video understanding benchmark,

    K. Li, Y . Wang, Y . He, Y . Li, Y . Wang, Y . Liu, Z. Wang, J. Xu, G. Chen, P. Luoet al., “Mvbench: A comprehensive multi-modal video understanding benchmark,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22 195– 22 206

    2024

  20. [28]

    Tempcompass: Do video llms really understand videos?

    Y . Liu, S. Li, Y . Liu, Y . Wang, S. Ren, L. Li, S. Chen, X. Sun, and L. Hou, “Tempcompass: Do video llms really understand videos?”arXiv preprint arXiv:2403.00476, 2024

    2024 arXiv

  21. [29]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis,

    C. Fu, Y . Dai, Y . Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y . Shen, M. Zhanget al., “Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis,”arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  22. [30]

    Llama-vid: An image is worth 2 tokens in large language models,

    Y . Li, C. Wang, and J. Jia, “Llama-vid: An image is worth 2 tokens in large language models,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 323–340

    2024

  23. [31]

    Long context transfer from language to vision,

    P. Zhang, K. Zhang, B. Li, G. Zeng, J. Yang, Y . Zhang, Z. Wang, H. Tan, C. Li, and Z. Liu, “Long context transfer from language to vision,”arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  24. [32]

    Unhackable temporal rewarding for scalable video mllms,

    E. Yu, K. Lin, L. Zhao, Y . Wei, Z. Zhu, H. Wei, J. Sun, Z. Ge, X. Zhang, J. Wanget al., “Unhackable temporal rewarding for scalable video mllms,”arXiv preprint arXiv:2502.12081, 2025

    2025 arXiv

  25. [33]

    Kangaroo: A powerful video-language model supporting long-context video input,

    J. Liu, Y . Wang, H. Ma, X. Wu, X. Ma, X. Wei, J. Jiao, E. Wu, and J. Hu, “Kangaroo: A powerful video-language model supporting long-context video input,”arXiv preprint arXiv:2408.15542, 2024

    2024 arXiv

  26. [34]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tanget al., “Qwen2. 5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  27. [35]

    Star: A benchmark for situated reasoning in real-world videos,

    B. Wu, S. Yu, Z. Chen, J. B. Tenenbaum, and C. Gan, “Star: A benchmark for situated reasoning in real-world videos,”arXiv preprint arXiv:2405.09711, 2024. 11

    2024 arXiv

  28. [36]

    The unreasonable effectiveness of entropy minimization in llm reasoning,

    S. Agarwal, Z. Zhang, L. Yuan, J. Han, and H. Peng, “The unreasonable effectiveness of entropy minimization in llm reasoning,”arXiv preprint arXiv:2505.15134, 2025

    2025 arXiv

  29. [37]

    Right question is already half the answer: Fully unsupervised llm reasoning incentivization,

    Q. Zhang, H. Wu, C. Zhang, P. Zhao, and Y . Bian, “Right question is already half the answer: Fully unsupervised llm reasoning incentivization,”arXiv preprint arXiv:2504.05812, 2025

    2025 arXiv

  30. [38]

    Learning to reason without external rewards,

    X. Zhao, Z. Kang, A. Feng, S. Levine, and D. Song, “Learning to reason without external rewards,”arXiv preprint arXiv:2505.19590, 2025

    2025 arXiv

  31. [39]

    Scalable best-of-n selection for large language models via self-certainty,

    Z. Kang, X. Zhao, and D. Song, “Scalable best-of-n selection for large language models via self-certainty,”arXiv preprint arXiv:2502.18581, 2025

    2025

  32. [40]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  33. [41]

    Trl: Transformer reinforcement learning,

    L. von Werra, Y . Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec, “Trl: Transformer reinforcement learning,” https://github.com/huggingface/ trl, 2020. 12 Part I Appendix Table of Contents A Discussion on Entropy-based Measureme...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.