Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RL-trained keyframe picker beats specialized video segmenters

desk verdict The two-system RL idea has real video-side evidence, but the RefAVS benchmark training contradicts the method, so the audio-visual headline claim is unsupported. read the letter →

arxiv 2505.20256 v1 pith:5EE7TSSM submitted 2025-05-26 cs.CV

Hao Zhong , Muzhi Zhu , Zongze Du , Zheng Huang , Canyu Zhao , Mingyu Liu , Wen Wang , Hao Chen
show 1 more author
Chunhua Shen
This is my paper · ORCID
classification cs.CV
keywords reinforcementlearningomnimodalreasoningvideoobjectsegmentationkeyframeselectionreferringaudio-visualGRPOtwo-systemarchitecturetaskreformulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the bottleneck in video-and-audio reasoning is not fine-grained perception but knowing which moments to look at and how to restate the question, and that this selection can be learned by reinforcement rather than hand-labeled supervision. It proposes Omni-R1, a two-system setup in which a low-resolution global reasoning model chooses keyframes and rewrites the user instruction, while a high-resolution detail model grounds and segments those snippets. The keyframes and rewrites are trained end-to-end with GRPO using rewards that measure keyframe quality, instruction-frame alignment, and final mask overlap against ground truth. With only one epoch of RL on a few thousand samples, the authors report that the trained selector surpasses their base model and specialized state-of-the-art video-segmentation systems on referring audio-visual and reasoning video segmentation, and also improves out-of-domain question-answering benchmarks.

What carries the argument

The load-bearing object is the GRPO-trained Global Reasoning System (System 1), a Qwen2.5-Omni-7B policy that outputs JSON segments of the form start_time, end_time, and description. It is trained with a hierarchical reward R = α_K R_K + α_A R_A + α_G R_G, where R_K rewards keyframe quality through temporal diversity, frame count near target, and normalized ground-truth mask area; R_A rewards frame-instruction alignment via Hungarian matching between grounded boxes and ground-truth boxes; and R_G rewards global temporal consistency as the average per-frame IoU between SAM2's propagated masks and ground-truth masks. These rewards are computed online through a frozen copy of the same model acting as the reference policy and detail system, providing dense, decomposable credit assignment for the otherwise ambiguous action of choosing keyframes and rewriting tasks.

What would settle it

Take the same Omni-R1 pipeline and replace the ground-truth-mask rewards R_G and R_saliency with rewards computed without ground-truth masks, such as instruction-grounding consistency only, or evaluate on a video benchmark where the target object is largest before the decisive action and occluded afterward; if J&F collapses or the policy still picks the largest-object frames, the GT-derived dense reward, not learned reasoning, was carrying the result.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that a reinforcement-learned System 1 resolves the temporal-spatial trade-off: instead of feeding every frame at high resolution, the model learns to pick a few temporally spread, object-salient keyframes and to rewrite the original long-horizon instruction into concrete visual descriptions that a frozen grounding-plus-SAM2 pipeline can execute. Training for one epoch on small task splits lifts performance well beyond the same-backbone baseline and beyond specialized models such as Sa2VA on RefAVS and ReVOS, with the largest gains on unseen splits. A further claimed effect is that this RL training improves out-of-domain generalization on OmniBench, VideoMME, and MVBench while supervised fine-tuning typically weakens it, and that joint audio-visual plus video training reduces audio-visual hallucination on AVHBench.

Load-bearing premise

The load-bearing premise is that the dense per-frame mask-overlap reward, computed through a frozen segmenter against ground-truth masks, is a stable and unbiased training signal for keyframe selection, and that selecting frames where the target object occupies the largest area is a valid proxy for keyframe informativeness; the paper's own appendix shows the audio-visual task had to be simplified to detection-only training because full audio with multi-frame video caused severe cross-modal hallucination.

Editorial extensions

If this is right

  • A single RL-trained selector can be paired zero-shot with different detail systems, so stronger perception modules can be swapped in without retraining System 1.
  • Since one epoch on a few thousand samples suffices, the training recipe is cheap enough to reproduce and scale to more task splits.
  • Reinforcement learning here improves out-of-domain generalization on general video and omni-modal QA benchmarks, in contrast to supervised fine-tuning which often degrades it.
  • The keyframe-and-reformulation policy transfers across referring audio-visual segmentation, reasoning video object segmentation, and general video QA, suggesting a common temporal-reasoning skill.
  • Joint training on audio-visual and video-only tasks reduces cross-modal hallucination relative to the base model, according to the AVHBench JUDGE subset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The method's gains may be partly attributable to the reward's use of ground-truth masks, which teaches the selector to maximize target visibility rather than to reason about narrative or causal cues; a benchmark that separates 'largest object' from 'task-critical moment' would test whether true temporal reasoning was learned.
  • Inference: Because System 2 is frozen and the reward is computed through SAM2, the approach is a form of reward shaping for a planning policy; the same recipe could apply to other perception backbones, such as detection or captioning, wherever a frozen downstream model can score the policy's output.
  • Inference: The paper's simplification of the audio-visual task to detection-only training suggests that true audio-visual temporal grounding remains unstable; extending the framework to full audio-visual segmentation may require a different reward decomposition than mask IoU.
  • Inference: The generalization gains on QA benchmarks raise a testable possibility: RL on segmentation tasks may teach a general 'think-before-answer' habit, which could be measured by probing whether the model's reasoning tokens become more informative after RL.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Omni-R1, a two-system architecture for omnimodal video-audio reasoning: System 1 (a global reasoning model) selects keyframes and rewrites the task, and System 2 (a detail-understanding model plus SAM2) performs pixel-level grounding on the selected high-resolution frames. The authors train System 1 with GRPO using hierarchical rewards (keyframe quality, frame-instruction alignment, and global temporal consistency) and report gains on RefAVS, ReVOS, MeViS, and general video/omnimodal QA benchmarks, including out-of-domain generalization improvements and reduced hallucination on AVHBench. The strongest evidence is the matched-backbone comparison in Table 1 and the REVOS results in Table 2, but the manuscript contains a major discrepancy between the described training method and what was actually trained for the audio-visual benchmark, and the reward ablation in Table 4 contradicts the formal reward objective in Section 3.3.

Significance. If the claims held as stated, the paper would be a meaningful contribution: it demonstrates that RL can train a keyframe selector and task rewriter for video segmentation with only one epoch on small datasets, and the matched-backbone gains over Qwen2.5-Omni-7B(SFT) with the same SAM2 pipeline are substantial. The paper also ships several useful diagnostics: the reward-component ablation (Table 4), resolution and thinking-prompt analysis (Table 7), and hallucination analysis on AVHBench (Table 6), plus released code. However, the RefAID training variant in Appendix B means the only audio-visual evidence does not exercise the proposed two-system keyframe-selection RL as described, and the ablation shows the full three-term reward is worse than a two-term subset. These issues are load-bearing for the paper's central omnimodal and hierarchical-reward claims, so the paper cannot be accepted in its current form.

major comments (3)
  1. [Appendix B, Section 4.2, Table 1] The audio-visual benchmark used to support the omnimodal claim is not trained with the method described. Appendix B ('Training and Inference Strategy') states that for RefAVS tasks, severe cross-modal hallucination led to a simplified variant RefAID that uses only the first video frame and the corresponding full audio query, with no SAM2 segmentation and training driven solely by detection-based rewards. Consequently, Equations (3)-(4) and the Section 3.3 reward hierarchy, including the global temporal consistency reward RG, do not apply to the RefAVS training runs that produce the Table 1 gains. Table 1's +16.4 and +8.0 J&F improvements are presented in Sections 1 and 4.2 as evidence for the two-system keyframe-selection RL framework, but no RefAVS experiment in the paper actually exercises temporal keyframe selection together with audio input and SAM2-based rewards. Please either report full audio-visual results using the method as described, or substantially revise the omnimodal claims so that they are restricted to the RefAID variant and to the video-only REVOS evidence; the current text conflates the two settings.
  2. [Section 3.3, Eq. (4), Table 4] The central reward specification is contradicted by the ablation that is supposed to support it. Table 4 reports ReVOS overall 39.9 for RK+RG and 38.4 for RK+RA+RG, so adding RA, one of the three hierarchical rewards in Eq. (4), hurts performance on the main benchmark. The hypothesis in Section C does not repair the problem: Section 3.3 presents R = αK RK + αA RA + αG RG as the training objective, and Section 4.2 attributes Omni-R1's gains to the hierarchical reward framework without stating which reward combination was used for the headline numbers. Please report the exact reward weights and the reward combination used for every reported model, and, if the final model excludes RA or uses different weights, revise Section 3.3 and Eq. (4) to match the experiments rather than presenting the full three-term reward as the method.
  3. [Section 1, Section 4.3, Table 3] The out-of-domain generalization claim ('reinforcement learning improves out-of-domain generalization, whereas conventional supervised fine-tuning often weakens it') is not established by the reported comparisons. Table 1 compares Omni-R1-7B with Qwen2.5-Omni-7B(SFT) only on RefAVS, whose training used the RefAID variant rather than the full RL pipeline; Tables 3-7 show gains over the base model but include models that were trained on AVS data (which, per Appendix B, is a detection-only variant) and do not control for the additional RefCOCOg grounding data or for the thinking prompt. A controlled comparison that isolates the RL-trained System 1 from SFT, with identical data and prompts, is needed before claiming that RL specifically improves out-of-domain generalization; as written, the claim is broader than the experiments support.
minor comments (6)
  1. [Section 4.1] The resolution notation '128×28×28' is unclear; please specify the actual video frame size, the number of input frames, and how the three factors correspond to tokens, patches, and channels.
  2. [Section 3.3, Eq. (4)] The subscript in αK,A,R appears to be a typo for αG; please define all reward weights explicitly and report their values.
  3. [Figure 6] Subcaptions (b) and (c) are copied from (a) and both refer to the 'sand pit' example, even though the figures show a violin/cello example and a rhythm example; the captions should describe the actual content.
  4. [References] The RefCOCOg dataset is cited as [41] in Section 4.1 but as [42] in Section C; please reconcile the citation numbering.
  5. [Abstract and Section 2.2] The claim of 'the first successful application of RL to large-scale omnimodal reasoning' should be softened or scoped, since Section 2.2 itself cites R1-Omni as concurrent work that also includes audio in RL training.
  6. [Table 1] The Qwen2.5-Omni-7B row shows a large gap between seen (31.6) and unseen (62.3) J&F that is not explained in the text; please clarify whether this reflects the evaluation protocol or a typo.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the RL-trained keyframe-selection claims are tested against external benchmarks, and the Appendix B RefAID substitution is an internal-validity concern rather than a circular derivation.

full rationale

The paper's derivation chain is an empirical RL pipeline, not a formal derivation. System 1 is trained with hierarchical rewards (R_K, R_A, R_G) that are direct functions of ground-truth masks and boxes, but using ground-truth labels as training rewards is standard supervised/RL practice and does not make the reported test-set predictions equivalent to the training inputs by construction. Table 1 and Table 2 report J&F on RefAVSBench and ReVOS, which are externally defined metrics evaluated on held-out or unseen splits; these metrics are not identical to the reward functions in Eqs. (2)-(4). No equation in the paper equates a prediction to a fitted input. There are no load-bearing self-citations: the external components (GRPO from DeepSeek-R1, SAM2, Qwen grounding, Sa2VA) are independently published and not by the present authors, and no uniqueness theorem is invoked. The one in-scope limitation worth flagging is Appendix B: 'For RefA VS tasks, we observed severe cross-modal hallucination issues during preliminary experiments... we introduce a simplified variant, RefAID... no SAM2 segmentation is used; training is driven solely by detection-based rewards.' This means the RefAVS results are not a clean test of the full temporal keyframe-selection mechanism described in Sections 3.1-3.3. That is a serious internal-validity/attribution problem for the omnimodal claim, but it is not a circularity in the derivation-chain sense: the RefAVS outcome is not equal to its training input by construction, nor is any parameter fitted to the target quantity and renamed as a prediction. The REVOS and general-QA results remain independent external tests. Hence the circularity score is 0, with the RefAID issue recorded as a non-circular correctness risk.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the framework composes existing models (Qwen2.5-Omni, SAM2) with a novel training scheme. The central claim rests on four assumptions: the validity of GRPO for multimodal policies, the reliability of the frozen System 2 reward channel, the use of GT mask area as a keyframe-quality proxy, and the solvability of the reformulated local queries. The GT-derived rewards and the unstated weights are the main hand-fit elements.

free parameters (3)
  • Reward weights alpha_K, alpha_A, alpha_G, lambda_1, lambda_2, lambda_3 = not reported
    Weights in R = alpha_K*RK + alpha_A*RA + alpha_G*RG and RK = lambda_1*Rdiversity + lambda_2*Rnum + lambda_3*Rsaliency (Section 3.3) are never given numeric values; the ablation in Table 4 shows the composition changes results by several points, so the exact training objective is under-specified.
  • Target keyframe count K0 = 4 (prompt: 'Select about 4 most relevant moments')
    Frame Count Regularization Rnum penalizes deviation from a predefined K0, which is chosen by hand in the Figure 3 prompt and not treated as a tunable hyperparameter in the paper.
  • Policy KL coefficient beta = 0.04 default, 0.001 in an ablation
    Reported in Section 4.1; Table 6 shows the AVHBench score changes from 60.8% to 61.5% when beta is reduced, indicating sensitivity to this hand-set value.
assumptions (4)
  • standard math GRPO/PPO advantage normalization (Eq. 1) provides valid policy-gradient updates for a multimodal policy.
    Imported from DeepSeek-R1 (Refs 17, 18); the paper assumes it transfers from language policies to video-audio policies without modification.
  • domain assumption Frozen System 2 (Qwen2.5-Omni grounding plus SAM2-hiera-large) yields reliable reward signals for RL.
    RG is computed through this frozen pipeline (Section 3.3); if SAM2 or the grounding model fails, the reward gradient is noise. Appendix B documents exactly such a failure for full-length RefAVS audio-video inputs and works around it with a detection-only variant.
  • domain assumption Ground-truth mask area is a valid proxy for keyframe informativeness.
    Rsaliency is defined as the normalized average GT mask area of selected frames (Section 3.3); this bakes the benchmark's annotation definition into the reward and may encourage selecting the largest object rather than the most informative moment.
  • domain assumption The two-system decomposition assumes reformulated local queries are solvable without temporal or audio context.
    System 2 sees only selected high-resolution frames (Section 3.1); the paper's own Limitations section (Appendix F) states System 2 lacks temporal context, which limits temporally sensitive tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration." pith.science (2026). https://pith.science/paper/5EE7TSSM

@misc{pith2026250520256,
  author       = {Pith},
  title        = {Pith review of: Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5EE7TSSM}},
  note         = {Machine review of arXiv:2505.20256}
}
read the original abstract

Long-horizon video-audio reasoning and fine-grained pixel understanding impose conflicting requirements on omnimodal models: dense temporal coverage demands many low-resolution frames, whereas precise grounding calls for high-resolution inputs. We tackle this trade-off with a two-system architecture: a Global Reasoning System selects informative keyframes and rewrites the task at low spatial cost, while a Detail Understanding System performs pixel-level grounding on the selected high-resolution snippets. Because ``optimal'' keyframe selection and reformulation are ambiguous and hard to supervise, we formulate them as a reinforcement learning (RL) problem and present Omni-R1, an end-to-end RL framework built on Group Relative Policy Optimization. Omni-R1 trains the Global Reasoning System through hierarchical rewards obtained via online collaboration with the Detail Understanding System, requiring only one epoch of RL on small task splits. Experiments on two challenging benchmarks, namely Referring Audio-Visual Segmentation (RefAVS) and Reasoning Video Object Segmentation (REVOS), show that Omni-R1 not only surpasses strong supervised baselines but also outperforms specialized state-of-the-art models, while substantially improving out-of-domain generalization and mitigating multimodal hallucination. Our results demonstrate the first successful application of RL to large-scale omnimodal reasoning and highlight a scalable path toward universally foundation models.

Figures

Figures reproduced from arXiv: 2505.20256 by the authors.

Figure 1
Figure 1. Overview of the proposed Omni-R1 system for collaborative video understanding. Left: Performance comparison across multiple benchmarks shows Omni-R1 significantly outperforms existing omni-modal and video-reinforced MLLMs on both segmentation-centric and reasoning￾centric tasks. Top-right: Omni-R1 employs a two-stage collaborative framework, integrating a detail understanding system (for precise visual QA) and a glo… view at source ↗
Figure 2
Figure 2. Exclusively trained as System 1 on video segmentation tasks in an End-to-End RL pipeline, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Keyframe selection and recaptioning prompt for System 1. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Audio analyzing and recaptioning prompt for System 1. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Visualization results of our method on three representative VOS cases. Each subfigure [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: More visualization results of our method on representative VOS and AVS cases. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Audio-Cogito: Towards Deep Audio Reasoning in Large Audio Language Models

    eess.AS 2026-04 unverdicted novelty 6.0 of 10

    Self-distillation of Qwen3-Omni-Thinking on 545k Cogito-Pipe audio reasoning traces yields the best open-source MMAR CoT scores and top-tier challenge ranking.

  2. Group Relative Policy Optimization for Speech Recognition

    eess.AS 2025-09 conditional novelty 5.0 of 10

    Applying GRPO with rule-based rewards to LLM-based ASR improves WER by up to 18.4% relative and reduces hallucination errors on unseen acoustic conditions.

Reference graph

Works this paper leans on

55 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    Baichuan-omni-1.5 technical report.arXiv preprint arXiv:2501.15368, 2025

    Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. Baichuan-omni-1.5 technical report.arXiv preprint arXiv:2501.15368, 2025. 1, 4, 9

    arXiv 2025

  2. [2]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024. 1, 3, 4, 9

    arXiv 2024

  3. [3]

    Attention-based multimodal fusion for video description

    Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi. Attention-based multimodal fusion for video description. In Proceedings of the IEEE international conference on computer vision, pages 4193–4202, 2017. 1

    work page 2017

  4. [4]

    Merlot reserve: Neural script knowledge through vision and language and sound

    Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375–16387, 2022. 1

    work page 2022

  5. [5]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 1

    arXiv 2024

  6. [6]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024. 1

    arXiv 2024

  7. [8]

    From seconds to hours: Reviewing multimodal large language models on comprehensive long video understanding.arXiv preprint arXiv:2409.18938, 2024

    Heqing Zou, Tianze Luo, Guiyang Xie, Fengmao Lv, Guangcong Wang, Junyang Chen, Zhuochen Wang, Hansheng Zhang, Huaijian Zhang, et al. From seconds to hours: Reviewing multimodal large language models on comprehensive long video understanding.arXiv preprint arXiv:2409.18938, 2024. 1

    arXiv 2024

  8. [9]

    Mavors: Multi-granularity video representation for multimodal large language model.arXiv preprint arXiv:2504.10068, 2025

    Yang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu, Yuanxing Zhang, Zihao Wang, Weihong Lin, Jingyun Hua, Zekun Wang, Xinlong Chen, et al. Mavors: Multi-granularity video representation for multimodal large language model.arXiv preprint arXiv:2504.10068, 2025. 1

    arXiv 2025

Show all 55 references
  1. [10]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024. 1, 9

    2024

  2. [11]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923,

    arXiv

  3. [12]

    Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023. 3

    2023 arXiv

  4. [13]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding.arXiv preprint arXiv:2306.17107, 2023

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding.arXiv preprint arXiv:2306.17107, 2023. 3

    2023 arXiv

  5. [14]

    Textmonkey: An ocr-free large multimodal model for understanding document.arXiv preprint arXiv:2403.04473, 2024

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document.arXiv preprint arXiv:2403.04473, 2024. 3

    2024 arXiv

  6. [15]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  7. [16]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4818–...

    2024

  8. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 3, 4

    2025 arXiv

  9. [18]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 3, 4

    2024 arXiv

  10. [19]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  11. [20]

    Omnibind: Large-scale omni multimodal representation via binding spaces.arXiv preprint arXiv:2407.11895, 2024

    Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Hengshuang Zhao, and Zhou Zhao. Omnibind: Large-scale omni multimodal representation via binding spaces.arXiv preprint arXiv:2407.11895, 2024. 3

    2024 arXiv

  12. [21]

    Ref- avs: Refer and segment objects in audio-visual scenes

    Yaoting Wang, Peiwen Sun, Dongzhan Zhou, Guangyao Li, Honggang Zhang, and Di Hu. Ref- avs: Refer and segment objects in audio-visual scenes. InEuropean Conference on Computer Vision, pages 196–213. Springer, 2024. 3, 8

    2024

  13. [22]

    Visa: Reasoning video object segmentation via large language models

    Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. arXiv preprint arXiv:2407.11325, 2024. 3, 8, 9

    2024 arXiv

  14. [23]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 4

    2023 arXiv

  15. [24]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023. 4

    2023 arXiv

  16. [25]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024. 4

    2024 arXiv

  17. [26]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 4

    2023 arXiv

  18. [27]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271,

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271,

    arXiv

  19. [28]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 4

    2023

  20. [29]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 4

    2023

  21. [30]

    Deepseek-vl: Towards real-world vision-language understanding, 2024

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision-language understanding, 2024. 4

    2024

  22. [31]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report.arXiv preprint arXiv:2503.20215,

    arXiv

  23. [32]

    Omnibench: Towards the future of universal omni-language models, 2024

    Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, and Chenghua Lin. Omnibench: To...

    2024

  24. [33]

    Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785,

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785,

    arXiv

  25. [34]

    Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025. 4

    2025 arXiv

  26. [35]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025. 4, 9

    2025 arXiv

  27. [36]

    Videochat-r1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning.arXiv preprint arXiv:2504.06958, 2025

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning.arXiv preprint arXiv:2504.06958, 2025. 4, 9, 10

    2025 arXiv

  28. [37]

    R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning, 2025

    Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning, 2025. 4

    2025

  29. [38]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. 5, 8

    2024 arXiv

  30. [39]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. InEuropean conference on computer vision, pages 213–229. Springer, 2020. 7, 12

    2020

  31. [40]

    MeViS: A large-scale benchmark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. MeViS: A large-scale benchmark for video segmentation with motion expressions. InICCV, 2023. 8

    2023

  32. [41]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. 2016. 8

    2016

  33. [42]

    Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement.arXiv preprint arXiv:2503.06520, 2025

    Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement.arXiv preprint arXiv:2503.06520, 2025. 8, 14

    2025 arXiv

  34. [43]

    Avsbench: A pixel-level audio- visual segmentation benchmark

    Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Avsbench: A pixel-level audio- visual segmentation benchmark. 8

  35. [44]

    Avsegformer: Audio-visual segmentation with transformer

    Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. Avsegformer: Audio-visual segmentation with transformer. InProceedings of the AAAI conference on artificial intelligence, volume 38, pages 12155–12163, 2024. 8

    2024

  36. [45]

    Prompting segmentation with sound is generalizable audio-visual source localizer

    Yaoting Wang, Weisong Liu, Guangyao Li, Jian Ding, Di Hu, and Xi Li. Prompting segmentation with sound is generalizable audio-visual source localizer. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5669–5677, 2024. 8

    2024

  37. [46]

    Language as queries for referring video object segmentation

    Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4974–4984, 2022. 8

    2022

  38. [47]

    Robust referring video object segmentation with cyclic structural consensus

    Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj, and Yan Lu. Robust referring video object segmentation with cyclic structural consensus. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 22236–22245, 2023. 8

    2023

  39. [48]

    Trackgpt–a generative pre-trained transformer for cross-domain entity trajectory forecasting.arXiv preprint arXiv:2402.00066, 2024

    Nicholas Stroh. Trackgpt–a generative pre-trained transformer for cross-domain entity trajectory forecasting.arXiv preprint arXiv:2402.00066, 2024. 9 22

    2024 arXiv

  40. [49]

    Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos.arXiv preprint arXiv:2501.04001, 2025

    Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos.arXiv preprint arXiv:2501.04001, 2025. 9, 15

    2025 arXiv

  41. [50]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024. 9

    2024 arXiv

  42. [51]

    Kangaroo: A powerful video-language model supporting long-context video input.arXiv preprint arXiv:2408.15542, 2024

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input.arXiv preprint arXiv:2408.15542, 2024. 9

    2024 arXiv

  43. [52]

    Vita-1.5: Towards gpt-4o level real-time vision and speech interaction.arXiv preprint arXiv:2501.01957, 2025

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction.arXiv preprint arXiv:2501.01957, 2025. 9

    2025 arXiv

  44. [53]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075,

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075,

    arXiv

  45. [54]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–222...

    2024

  46. [55]

    David F. Crouse. On implementing 2d rectangular assignment algorithms.IEEE Transactions on Aerospace and Electronic Systems, 52(4):1679–1696, 2016. 12

    2016

  47. [56]

    Avhbench: A cross-modal hallucination benchmark for audio-visual large language models

    Kim Sung-Bin, Oh Hyun-Bin, JungMok Lee, Arda Senocak, Joon Son Chung, and Tae-Hyun Oh. Avhbench: A cross-modal hallucination benchmark for audio-visual large language models. arXiv preprint arXiv:2410.18325, 2024. 15 23

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.