Pith. sign in

REVIEW 3 major objections 4 minor 6 cited by

HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Summarizing multimodal context before reasoning lifts scores across three benchmarks, the paper reports.

desk verdict The context-before-reasoning recipe is a real, simple contribution with credible out-of-domain gains, but the IntentBench headline is compromised by a likely train/test overlap that the authors must address. read the letter →

arxiv 2506.21277 v1 pith:27WCFWZG submitted 2025-06-26 cs.CV cs.CL

Qize Yang , Shimin Yao , Weixuan Chen , Shenghao Fu , Detao Bai , Jiaxing Zhao , Boyuan Sun , Bowen Yin
show 2 more authors
Xihan Wei Jingren Zhou
This is my paper
classification cs.CVcs.CL
keywords omni-modalreasoningreinforcementlearningcontextgroundinghumanintentionunderstandingmultimodalbenchmarkvideo-audioGRPOLLM-as-a-judgerewards
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that omni-modal reasoning models fail not for lack of compute but because they either misread the global multimodal context or skip it entirely, jumping from question to answer on textual patterns. HumanOmniV2 is a 7B model trained to emit an explicit context summary of the video-and-audio evidence before it reasons, and reinforcement-learning rewards are designed to make that summary accurate and the reasoning logically grounded in it. On three audio-visual reasoning benchmarks the model reports the best open-source scores, including 69.33% on IntentBench, a new benchmark that asks about human intentions, emotions, and deception. If correct, this is evidence that explicit context grounding, not larger scale, is a main lever for multimodal reasoning in human-centric settings.

What carries the argument

The load-bearing mechanism is a context-summarization stage wrapped in a <context> tag, generated before the <think> reasoning block, and two LLM-judged rewards applied only to the tokens of the part they evaluate: a context reward comparing the generated context to a reference context on a 0-5 coverage scale, and a logical reward scoring whether the reasoning integrates multimodal evidence, reflects, confirms, and deduces without contradictions. These join a binary format reward and rule-based accuracy rewards inside a GRPO objective with token-level loss, no question normalization, and dynamic KL. The paper also contributes IntentBench, 633 videos and 2,689 questions from Social-IQ 2.0, EMER, and MDPE, curated so every question needs both audio and video.

What would settle it

Take the trained model, hold out a sample of IntentBench videos, and have humans rate each generated context summary for factual accuracy against the video and audio; if summaries that earn high context-reward scores frequently contain details not present in the input, the reward is measuring fluency rather than understanding. A cheaper check is to re-run training with the context reward scores randomly shuffled; the reported IntentBench gap should collapse if the reward carries the signal.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that requiring a multimodal model to first restate the global context of an input, and then rewarding that restatement and the reasoning built on it separately, substantially reduces two failure modes: insufficient global context understanding and shortcut answers that ignore video or audio. The model uses a context-think-answer format; an LLM judge scores the context for coverage of a reference context and scores the reasoning for integration of multimodal clues, reflection, and logical deduction, and these scores become masked token-level rewards in GRPO. The paper reports 58.47% on Daily-Omni, 47.1% on WorldSense, and 69.33% on IntentBench, all above previous open-source omni-modal models.

Load-bearing premise

The whole reward pipeline rests on trusting an LLM judge's 0-5 scores for context coverage and logical quality, and on the unstated rule that turns those scores into binary training rewards $r_c$ and $r_l$, so the model could in principle be rewarded for verbose, plausible-sounding summaries rather than genuine multimodal understanding.

Editorial extensions

If this is right

  • Out-of-domain gains: HumanOmniV2 scores 58.47% on Daily-Omni and 47.1% on WorldSense, ahead of other open-source omni-modal models, including a jump from 61.71 to 74.28 on Daily-Omni's reasoning task.
  • In-domain results: the full method reaches 69.33% on IntentBench, beating the SFT cold-start baseline (62.03%), the method without context and logical rewards (68.44%), and Qwen2.5-Omni (64.20%).
  • The context-think-answer format is a general recipe: any omni-modal base model can be prompted and rewarded the same way without architectural changes.
  • IntentBench adds a reusable evaluation that requires simultaneous audio and video understanding for human intention, emotion, and deception, where prior benchmarks needed only one modality.
  • Two-stage RL training suggests the judge-based rewards are needed while the model is learning context and reasoning; stage 2 drops them and still preserves general capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the paper leaves implicit: if context grounding is what drives the gains, performance should degrade most on questions that pit audio against video evidence when the context tag is removed at inference time; that is directly testable on IntentBench.
  • A second inference: the LLM judge may be distilling its own reading of the video into the policy, so part of the reported gain could come from transferring the judge's interpretation rather than from the model's raw perception; comparing judge-scored rewards against purely rule-based rewards on the same data would separate the two.
  • The paper's own limitation suggests an extension: letting the model revise its context summary after reasoning, rather than fixing it up front, could recover corrections the current format blocks.
  • A practical extension would be publishing the threshold that maps 0-5 judge scores to binary rewards; without it, the recipe is not fully reproducible and reward hacking cannot be ruled out.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents HumanOmniV2, an omni-modal reasoning model built on Qwen2.5-Omni-7B-thinker. The method trains the model to first emit a <context> summary of the multimodal input, then reason in <think> and answer in <answer>, using GRPO with an accuracy reward, a format reward, an LLM-judged context reward, and an LLM-judged logical reward. The authors also introduce IntentBench, a 633-video, 2,689-question benchmark curated from Social-IQ 2.0, EMER, and MDPE, and report state-of-the-art open-source results on IntentBench (69.33%), Daily-Omni (58.47%), and WorldSense (47.1%). The paper includes cold-start SFT, two-stage RL, ablations on the context and logical rewards, and qualitative examples.

Significance. If the central claims hold, the paper makes a useful contribution: it proposes a concrete recipe for making omni-modal reasoning less shortcut-prone by requiring explicit global-context summarization before reasoning, and it introduces a benchmark focused on human intentions and emotions that require joint audio-visual understanding. The out-of-domain results on Daily-Omni and WorldSense are encouraging, and the qualitative examples illustrate the intended behavior. The release of code and the inclusion of ablation variants are assets. However, the headline in-domain result depends on the integrity of IntentBench, and the reward mechanism is under-specified; these issues currently prevent the paper from being accepted as is.

major comments (3)
  1. [§3 and §5.1, Table 3] The IntentBench test set is not shown to be disjoint from the RL/cold-start training data, and for EMER the paper's own numbers make disjointness impossible on the natural reading. Section 3 states that EMER has 332 video samples and that the authors "randomly select 133 videos" for IntentBench testing; Section 5.1 states that the training data include "the remaining 200 entries from EMER." Since 200 + 133 = 333 > 332, the two sets cannot be disjoint unless specific videos are excluded and disclosed, which the paper does not do. For Social-IQ 2.0, the paper never states that the 300 selected IntentBench videos come from a held-out split; Section 5.1 says the training data include "the training set of Social-IQ2.0," and Section 5.3 concedes that Qwen2.5-Omni may already have been trained on the original datasets. The Section 8.3 mitigation (rewriting options and GPT-4o filtering) reduces superficial text memorization but does not nullify an advantage from RL training on the same videos' audio-visual content, and EMER receives no such mitigation. Consequently, the reported 69.33% score and the Social and Emotion columns of Table 3 are not reliable evidence for the central claim. The authors must either prove disjointness with exact video identifiers or re-curate the test set and rerun all affected comparisons.
  2. [§4.3 and Figs. 11–12] The conversion from the LLM judge's 0–5 scores to the binary rewards rc and rl is never defined. The prompts in Figures 11 and 12 ask the LLM to "only return the score number," but the paper does not state which threshold maps a score to rc = 1 or rl = 1, nor whether the threshold was tuned. This makes the training signal non-reproducible and leaves open the possibility that the reward encourages verbose or stylistically convincing text rather than genuine multimodal understanding. The authors should specify the exact mapping, report the judge's agreement with human judgments on a sample, and ideally show sensitivity of the final accuracy to the threshold.
  3. [Table 3] The central ablation claim that the context and logical rewards help rests on small differences without any measure of variability: the full method scores 69.33, "Ours w/o ER" scores 68.44, and "Ours w/o context and ER" scores 66.72, with no repeated runs, seeds, or error bars. This is especially concerning because the evaluation set itself is potentially contaminated for the Social-IQ 2.0 and EMER portions. The authors should report multiple runs or, at minimum, verify the same ablation pattern on the out-of-domain Daily-Omni and WorldSense benchmarks, which are unaffected by the overlap concern.
minor comments (4)
  1. [Table 1] The MiniCPM-o entry in Table 1 is cited as [3], but reference [3] is Ola; the citation should likely be MiniCPM-o 2.6 [17] or a distinct reference for the model actually evaluated.
  2. [§3 and Fig. 12] The score rubric in Figure 12 lists 5, 4, 2, 1, and 0 points but omits the 3-point descriptor, even though 3 is a possible score; this makes the reward prompt incomplete as printed.
  3. [§3, Fig. 2] There is a typo in the figure text: "answersing" should be "answering." Also, the caption of Table 3 defines "ER" after the acronym is used; consider defining it in the main text before the table.
  4. [§8.3 and reproducibility] The paper does not state where the IntentBench test set, the curation scripts, or the exact split identifiers will be released; the GitHub link only describes the model. For a benchmark paper, releasing the benchmark artifacts and evaluation script is important for verification.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity in the IntentBench evaluation: the benchmark and the RL training set are drawn from the same Social-IQ 2.0 and EMER pools with no stated disjoint split, so the headline 69.33% is partly a re-measurement of training data; the method itself retains independent support from Daily-Omni and WorldSense.

  1. fitted input called prediction [Section 5.1 (training data) vs. Section 3 (IntentBench construction); see also Section 8.3]
    "we sample part of the data from Video-R1, the training set of Social-IQ2.0, and the remaining 200 entries from EMER. ... It includes 332 video samples from MER2023 ... We randomly select 133 videos and their corresponding questions as the testing set."

    The model is optimized on the same data sources that define IntentBench. For EMER the stated counts force overlap: 332 total videos cannot supply 200 RL-training entries plus 133 test videos (333 > 332) unless an exclusion is disclosed, and none is. For Social-IQ 2.0 the paper never states that its 300 benchmark videos come from a held-out split, while RL explicitly uses the training set; Section 5.3 concedes the base model may already have seen these datasets. The reported IntentBench performance is therefore not an independent out-of-sample prediction for the EMER and Social-IQ portions; it is a statistically forced re-measurement of training-distribution content. Daily-Omni and WorldSense are external and unaffected, so the circularity is partial.

full rationale

There is no equation-level self-definition: the GRPO objective (Sec. 4.1) is a standard policy-gradient update, and the LLM-judged context/logical rewards are training signals, not evaluation metrics. The listed self-citations (HumanOmni, R1-Omni, ViSpeak, ActionArt) are related-work only and are not load-bearing. The one substantive circularity is the construction of IntentBench from the same Social-IQ 2.0 and EMER sources used for RL training, with no demonstrated train/test disjointness; the EMER arithmetic (200+133 > 332) makes the partial overlap concrete. This inflates the in-domain Table 3 result but does not invalidate the method: Daily-Omni and WorldSense are external benchmarks, and the model improves there too. Hence a partial-circularity score of 4, not 6-10, is appropriate.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities. The most consequential free parameter is the unspecified mapping from LLM judge scores to binary rewards. The load-bearing assumptions are that context summarization helps, that LLM judges give trustworthy scores, and that IntentBench is a valid disjoint audio-visual benchmark.

free parameters (1)
  • LLM judge score-to-reward threshold = unspecified; paper defines r_c and r_l as binary but judge prompts return 0-5
    The conversion from the judge's 0-5 score to the binary reward used in GRPO is never stated. The threshold changes reward density and could materially affect training outcomes.
assumptions (3)
  • domain assumption Summarizing global multimodal context before reasoning reduces shortcut behavior and improves answer accuracy.
    This is the paper's core hypothesis, introduced in Section 4.2. It is tested empirically but not derived from first principles.
  • domain assumption LLM judges provide reliable, unbiased assessments of context coverage and logical reasoning quality.
    Context and logical rewards are computed from LLM judge scores (Section 4.3). If the judges are unreliable or gameable, the training signal is corrupted.
  • domain assumption IntentBench questions require joint audio-visual understanding and cannot be answered from text alone.
    The benchmark is curated with a GPT-4o text-only filter and manual verification (Section 3), but this is not a formal guarantee for every question.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context." pith.science (2026). https://pith.science/paper/27WCFWZG

@misc{pith2026250621277,
  author       = {Pith},
  title        = {Pith review of: HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27WCFWZG}},
  note         = {Machine review of arXiv:2506.21277}
}
read the original abstract

With the rapid evolution of multimodal large language models, the capacity to deeply understand and interpret human intentions has emerged as a critical capability, which demands detailed and thoughtful reasoning. In recent studies, Reinforcement Learning (RL) has demonstrated potential in enhancing the reasoning capabilities of Large Language Models (LLMs). Nonetheless, the challenges associated with adapting RL to multimodal data and formats remain largely unaddressed. In this paper, we identify two issues in existing multimodal reasoning models: insufficient global context understanding and shortcut problems. Insufficient context understanding can happen when a model misinterprets multimodal context, resulting in incorrect answers. The shortcut problem occurs when the model overlooks crucial clues in multimodal inputs, directly addressing the query without considering the multimodal information. To tackle these issues, we emphasize the necessity for the model to reason with a clear understanding of the global context within multimodal inputs. This global context understanding can effectively prevent the model from overlooking key multimodal cues and ensure a thorough reasoning process. To ensure the accurate interpretation of multimodal context information, we implement a context reward judged by a large language model, alongside format and accuracy rewards. Additionally, to improve complex reasoning capability, we employ the LLM to assess the logical reward, determining whether the reasoning process successfully integrates multimodal information with logical methods. We also introduce a reasoning omni-modal benchmark, IntentBench, aimed at evaluating models in understanding complex human intentions and emotions. Our proposed method demonstrates advanced performance across multiple omni-modal benchmarks compared to other open-source omni-modal models.

Figures

Figures reproduced from arXiv: 2506.21277 by the authors.

Figure 1
Figure 1. Visualizations of the vanilla GRPO method applied in multimodal tasks. When the model [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a)(b)(c) are examples from Social-IQ 2.0, MDPE, and EMER, respectively. (d) is the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The reasoning path of our model on an example from Social-IQ 2.0. The model first clearly [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Illustration of our method. We use Qwen2.5-Omni-Thinker[ [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Visualization result of our method on IntentBench. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Visualization result of our method and the compared method on IntentBench. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Visualization result of our method and the compared method on IntentBench. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Visualization result of our method and the compared method on IntentBench. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Visualization result of our method and the compared method on IntentBench. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: The system prompt of our model. Please analyze whether the reasoning text is derived from the evidence and context text based on the following criteria and give a score of 0-5: Grading criteria description (relevance and rationality): Integration of Clues (1 point): D…
Figure 11
Figure 11. Figure 11: The prompt for assessing the logical reward. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: The prompt for assessing the context reward. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Comparison of the original options in Social-IQ 2.0 and our refined options. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Comparison of the original open-vocabulary answer and our designed multi-choice [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A training-free two-stage token compression framework preserves 92.9% of Qwen2.5-Omni-7B's audio-visual understanding accuracy while using 6.8% of the original multimodal-token FLOPs.

  2. OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A training-free, query-anchored, modality-decoupled token compression method outperforms unidirectional audio/video compression baselines on Qwen2.5-Omni while cutting prefill cost.

  3. Empowering Long-form Omni-modal Understanding with Robust Audio Perception

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Decoupled audio-visual caption and CoT-QA datasets plus two-stage fine-tuning measurably strengthen auditory perception and cross-modal reasoning in a 7B omni-modal LLM.

  4. TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions

    cs.CV 2026-02 reject novelty 6.0 of 10

    TimeChat-Captioner introduces Omni Dense Captioning—continuous, timestamped, six-dimensional audio-visual scene captions—plus a benchmark, a matching metric (SodaM), and a 7B model trained with SFT+GRPO.

  5. EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs

    cs.CV 2025-12 conditional novelty 6.0 of 10

    EchoingPixels prunes audio-visual LLM input tokens jointly across modalities and re-tunes RoPE frequencies so that 5–20% of tokens retain roughly full-model performance.

  6. LenGuard-GPC: Length Guarding with Guided-Prompt Consistency for Spatial Reasoning Reinforce Learning

    cs.AI 2026-07 conditional novelty 5.0 of 10

    LenGuard-GPC adds a token-level KL consistency reward between standard and guided prompts, plus a staged length bonus, to GRPO training of Qwen3-VL-8B and reports better accuracy with shorter responses on multi-view s...

Reference graph

Works this paper leans on

49 extracted references · 6 canonical work pages · cited by 6 Pith papers

  1. [1]

    Qwen2. 5-omni technical report,

    J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y . Fan, K. Dang et al., “Qwen2. 5-omni technical report,” arXiv preprint arXiv:2503.20215, 2025

    arXiv 2025

  2. [2]

    Ocean-omni: To understand the world with omni-modality,

    Y . Li, H. Sun, M. Lin, T. Li, G. Dong, T. Zhang, B. Ding, W. Song, Z. Cheng, Y . Huoet al., “Ocean-omni: To understand the world with omni-modality,”arXiv preprint arXiv:2410.08565, 2024

    arXiv 2024

  3. [3]

    Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment,

    Z. Liu, Y . Dong, J. Wang, Z. Liu, W. Hu, J. Lu, and Y . Rao, “Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment,” arXiv preprint arXiv:2502.04328, 2025

    arXiv 2025

  4. [4]

    Vita-1.5: Towards gpt-4o level real-time vision and speech interaction,

    C. Fu, H. Lin, X. Wang, Y .-F. Zhang, Y . Shen, X. Liu, H. Cao, Z. Long, H. Gao, K. Liet al., “Vita-1.5: Towards gpt-4o level real-time vision and speech interaction,”arXiv preprint arXiv:2501.01957, 2025

    arXiv 2025

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Biet al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  6. [6]

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning,

    F. Meng, L. Du, Z. Liu, Z. Zhou, Q. Lu, D. Fu, T. Han, B. Shi, W. Wang, J. He et al., “Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning,” arXiv preprint arXiv:2503.07365, 2025

    arXiv 2025

  7. [8]

    Visual-rft: Visual reinforcement fine-tuning,

    Z. Liu, Z. Sun, Y . Zang, X. Dong, Y . Cao, H. Duan, D. Lin, and J. Wang, “Visual-rft: Visual reinforcement fine-tuning,” arXiv preprint arXiv:2503.01785, 2025

    arXiv 2025

  8. [9]

    Vlm-r1: A stable and generalizable r1-style large vision-language model,

    H. Shen, P. Liu, J. Li, C. Fang, Y . Ma, J. Liao, Q. Shen, Z. Zhang, K. Zhao, Q. Zhanget al., “Vlm-r1: A stable and generalizable r1-style large vision-language model,” arXiv preprint arXiv:2504.07615, 2025

    arXiv 2025

Show all 49 references
  1. [10]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,” arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  2. [11]

    Let’s verify step by step,

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” inThe Twelfth International Conference on Learning Representations, 2023

    2023

  3. [12]

    Deep reinforcement learning from human preferences,

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” Advances in neural information processing systems , vol. 30, 2017

    2017

  4. [13]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  5. [14]

    Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities,

    Z. Zhou, R. Wang, and Z. Wu, “Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities,” arXiv preprint arXiv:2505.17862, 2025

    2025

  6. [15]

    Worldsense: Evaluating real-world omnimodal understanding for multimodal llms,

    J. Hong, S. Yan, J. Cai, X. Jiang, Y . Hu, and W. Xie, “Worldsense: Evaluating real-world omnimodal understanding for multimodal llms,” arXiv preprint arXiv:2502.04326, 2025

    2025 arXiv

  7. [16]

    Hu- manomni: A large vision-speech language model for human-centric video understanding,

    J. Zhao, Q. Yang, Y . Peng, D. Bai, S. Yao, B. Sun, X. Chen, S. Fu, X. Wei, L. Bo et al. , “Hu- manomni: A large vision-speech language model for human-centric video understanding,” arXiv preprint arXiv:2501.15111, 2025

    2025 arXiv

  8. [17]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. Heet al., “Minicpm-v: A gpt-4v level mllm on your phone,” arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  9. [18]

    Internlm- xcomposer2. 5-omnilive: A comprehensive multimodal system for long-term streaming video and audio interactions,

    P. Zhang, X. Dong, Y . Cao, Y . Zang, R. Qian, X. Wei, L. Chen, Y . Li, J. Niu, S. Dinget al., “Internlm- xcomposer2. 5-omnilive: A comprehensive multimodal system for long-term streaming video and audio interactions,” arXiv preprint arXiv:2412.09596, 2024

    2024 arXiv

  10. [19]

    Vispeak: Visual instruction feedback in streaming videos,

    S. Fu, Q. Yang, Y .-M. Li, Y .-X. Peng, K.-Y . Lin, X. Wei, J.-F. Hu, X. Xie, and W.-S. Zheng, “Vispeak: Visual instruction feedback in streaming videos,” arXiv preprint arXiv:2503.12769, 2025

    2025 arXiv

  11. [20]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos,

    K. Hu, P. Wu, F. Pu, W. Xiao, Y . Zhang, X. Yue, B. Li, and Z. Liu, “Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos,” arXiv preprint arXiv:2501.13826, 2025

    2025 arXiv

  12. [21]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis,

    C. Fu, Y . Dai, Y . Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y . Shen, M. Zhanget al., “Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis,” in Proceedings of the Computer Vision and Pattern Recognition Conference , 2025, ...

    2025

  13. [22]

    Actionart: Advancing multi- modal large models for fine-grained human-centric video understanding,

    Y .-X. Peng, Q. Yang, Y .-M. Tang, S. Fu, K.-Y . Lin, X. Wei, and W.-S. Zheng, “Actionart: Advancing multi- modal large models for fine-grained human-centric video understanding,”arXiv preprint arXiv:2504.18152, 2025. 18

    2025 arXiv

  14. [23]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun et al. , “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  15. [24]

    Omnibench: Towards the future of universal omni-language models,

    Y . Li, G. Zhang, Y . Ma, R. Yuan, K. Zhu, H. Guo, Y . Liang, J. Liu, Z. Wang, J. Yanget al., “Omnibench: Towards the future of universal omni-language models,”arXiv preprint arXiv:2409.15272, 2024

    2024

  16. [25]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models,

    W. Huang, B. Jia, Z. Zhai, S. Cao, Z. Ye, F. Zhao, Z. Xu, Y . Hu, and S. Lin, “Vision-r1: Incentivizing reasoning capability in multimodal large language models,” arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  17. [26]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization,

    J. Zhang, J. Huang, H. Yao, S. Liu, X. Zhang, S. Lu, and D. Tao, “R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization,” arXiv preprint arXiv:2503.12937, 2025

    2025 arXiv

  18. [27]

    Insight-v: Exploring long-chain visual reasoning with multimodal large language models,

    Y . Dong, Z. Liu, H.-L. Sun, J. Yang, W. Hu, Y . Rao, and Z. Liu, “Insight-v: Exploring long-chain visual reasoning with multimodal large language models,” arXiv preprint arXiv:2411.14432, 2024

    2024 arXiv

  19. [28]

    Visionary-r1: Mitigating shortcuts in visual reasoning with reinforcement learning,

    J. Xia, Y . Zang, P. Gao, Y . Li, and K. Zhou, “Visionary-r1: Mitigating shortcuts in visual reasoning with reinforcement learning,” arXiv preprint arXiv:2505.14677, 2025

    2025

  20. [29]

    Observe-r1: Unlocking reasoning abilities of mllms with dynamic progressive reinforcement learning,

    Z. Guo, M. Hong, and T. Jin, “Observe-r1: Unlocking reasoning abilities of mllms with dynamic progressive reinforcement learning,” arXiv preprint arXiv:2505.12432, 2025

    2025 arXiv

  21. [30]

    R1-omni: Explainable omni-multimodal emotion recognition with reinforce- ment learning,

    J. Zhao, X. Wei, and L. Bo, “R1-omni: Explainable omni-multimodal emotion recognition with reinforce- ment learning,” arXiv preprint arXiv:2503.05379, 2025

    2025 arXiv

  22. [31]

    Echoink-r1: Exploring audio-visual reasoning in multimodal llms via reinforcement learning,

    Z. Xing, X. Hu, C.-W. Fu, W. Wang, J. Dai, and P.-A. Heng, “Echoink-r1: Exploring audio-visual reasoning in multimodal llms via reinforcement learning,” arXiv preprint arXiv:2505.04623, 2025

    2025 arXiv

  23. [32]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,” arXiv preprint arXiv:2310.02255, 2023

    arXiv

  24. [33]

    Mmvu: Measuring expert-level multi-discipline video understanding,

    Y . Zhao, L. Xie, H. Zhang, G. Gan, Y . Long, Z. Hu, T. Hu, W. Chen, C. Li, J. Song et al. , “Mmvu: Measuring expert-level multi-discipline video understanding,” arXiv preprint arXiv:2501.12380, 2025

    2025 arXiv

  25. [34]

    Social-iq: A question answering benchmark for artificial social intelligence,

    A. Zadeh, M. Chan, P. P. Liang, E. Tong, and L.-P. Morency, “Social-iq: A question answering benchmark for artificial social intelligence,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8807–8817

    2019

  26. [35]

    Social-iq 2.0 challenge: Benchmarking multimodal social understanding,

    A. Wilf, L. Mathur, S. Mathew, C. Ko, Y . Kebe, P. P. Liang, and L.-P. Morency, “Social-iq 2.0 challenge: Benchmarking multimodal social understanding,” https://github.com/abwilf/Social-IQ-2.0-Challenge, 2023

    2023

  27. [36]

    Explainable multimodal emotion recognition,

    Z. Lian, H. Sun, L. Sun, H. Gu, Z. Wen, S. Zhang, S. Chen, M. Xu, K. Xu, K. Chen et al., “Explainable multimodal emotion recognition,” arXiv preprint arXiv:2306.15401, 2023

    2023 arXiv

  28. [37]

    Mdpe: A multi- modal deception dataset with personality and emotional characteristics,

    C. Cai, S. Liang, X. Liu, K. Zhu, Z. Wen, J. Tao, H. Xie, J. Cui, Y . Ma, Z. Chenget al., “Mdpe: A multi- modal deception dataset with personality and emotional characteristics,” arXiv preprint arXiv:2407.12274, 2024

    2024 arXiv

  29. [39]

    Dapo: An open-source llm reinforcement learning system at scale,

    Q. Yu, Z. Zhang, R. Zhu, Y . Yuan, X. Zuo, Y . Yue, T. Fan, G. Liu, L. Liu, X. Liu et al., “Dapo: An open-source llm reinforcement learning system at scale,” arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  30. [40]

    Understanding r1-zero-like training: A critical perspective,

    Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin, “Understanding r1-zero-like training: A critical perspective,” arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  31. [41]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  32. [42]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” inText summarization branches out, 2004, pp. 74–81

    2004

  33. [43]

    Video-r1: Reinforcing video reasoning in mllms,

    K. Feng, K. Gong, B. Li, Z. Guo, Y . Wang, T. Peng, B. Wang, and X. Yue, “Video-r1: Reinforcing video reasoning in mllms,” arXiv preprint arXiv:2503.21776, 2025

    2025 arXiv

  34. [44]

    Gemini 2.5 pro,

    Google, “Gemini 2.5 pro,” https://deepmind.google/technologies/gemini/pro/, 2025

    2025

  35. [45]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action,

    J. Lu, C. Clark, S. Lee, Z. Zhang, S. Khosla, R. Marten, D. Hoiem, and A. Kembhavi, “Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. ...

    2024

  36. [46]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms,

    Z. Cheng, S. Leng, H. Zhang, Y . Xin, X. Li, G. Chen, Y . Zhu, W. Zhang, Z. Luo, D. Zhao et al. , “Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms,” arXiv preprint arXiv:2406.07476, 2024. 19

    2024 arXiv

  37. [47]

    Introducing the next generation of Claude,

    Anthropic, “Introducing the next generation of Claude,” https://www.anthropic.com/news/claude-3-family, 2024, accessed: 2024-10-22

    2024

  38. [48]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  39. [49]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wanget al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  40. [50]

    Openai o1 system card,

    A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney et al., “Openai o1 system card,” arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  41. [51]

    Affectgpt: Dataset and framework for explainable multimodal emotion recognition,

    Z. Lian, H. Sun, L. Sun, J. Yi, B. Liu, and J. Tao, “Affectgpt: Dataset and framework for explainable multimodal emotion recognition,” arXiv preprint arXiv:2407.07653, 2024. 20

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.