Pith. sign in

REVIEW 3 major objections 6 minor 109 references

Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single autoregressive objective can learn both when to speak and what to say, on a shared 80-millisecond clock linking video, audio, and text output.

desk verdict A genuinely useful duplex-streaming architecture with an honest limitations section, but the 'learns when to respond' claim is load-bearing on a hand-tuned silence masking ratio and lacks direct timing evidence. read the letter →

arxiv 2608.08469 v1 pith:3UKUOJ6N submitted 2026-08-09 cs.AI

Kaichen Zhang , Wei Huang , Keming Wu , Bo Li , Xiaojuan Qi This is my paper · ORCID
classification cs.AI
keywords streamingmultimodalgenerationduplexinteractionsilencetokenslot-alignedsupervisionrealtimevideounderstandinglow-latencyincrementalinferenceKV-cachereuseproactiveresponsemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that turn-based prefill-then-decode interaction is not required for realtime multimodal assistants. The proposed 4B system, Aero Realtime, places video, audio, and text output on one temporal grid of roughly 80-ms slots, and at each slot predicts either a lexical token or a dedicated silence token, so a single next-token loss learns both when to respond and what to generate. Because input and output advance on the same clock, the model can perceive while speaking, and because each update appends only the newest multimodal slot and reuses the key-value cache, a 20-minute continuous stream stays within about 200 ms of the source timeline on four workstation GPUs. These results matter because they point to a serving-friendly alternative to polling loops and external response gates for continuous, proactive interaction.

What carries the argument

The load-bearing mechanism is the slot-aligned input-output stream: each ~80-ms audio slot is fused with the preceding output-token embedding ($e_t^a = z_t^a + E(y_{t-1})$) to form the next causal state, and the shared language-model head predicts a lexical token or the silence token $[P]$ from that state. This single mechanism carries the whole argument: it admits new observations into an active generation sequence, makes silence and speech comparable outputs of one objective, and enables delta inference in which only the newest multimodal slot is evaluated while the KV cache keeps the prefix.

What would settle it

Run the same recipe at r=0.95 and r=0.90 on the same held-out stream and compare the delay between each response onset and the event it describes; if the onsets shift or vanish at r=0.90, the masking ratio, not the aligned objective, controls timing.

Watch

Extended reading notes

Core claim

The central claim is that response timing is a linguistic prediction problem, not a separate control problem. Aero Realtime couples every ~80-ms audio slot with one output slot whose target is either a lexical token or the special silence token [P]; the audio representation at slot t is added to the embedding of the previous output token, so each step is conditioned on both fresh perception and the model's own most recent output, while timestamped visual tokens enter the same causal sequence. Training remains standard next-token cross-entropy, and inference appends only the newly arrived multimodal slot while carrying the sampled token into the next fused state and preserving the KV-cached prefix. On OVOBench the 4B model scores 61.49 realtime, 44.07 backward, and 39.36 forward, comparable to several prior 7B methods, and in a long-horizon service test it maintains an 84-ms median and 173-ms P95 processing lag over the first 20 minutes of a streamed video, remaining within 200 ms of the source timeline.

Load-bearing premise

The load-bearing premise is that randomly dropping 95% of silence-target losses is a benign training detail: the Section 4.4 ablation shows that without this masking the model emits silence almost always and scores 6.40 on OVOBench, so the aligned objective alone is not demonstrated to learn when to respond.

Editorial extensions

If this is right

  • A duplex assistant can be trained with ordinary next-token supervision: no external response gate or polling controller is needed to decide when to speak.
  • Long-running interactions remain servable because each 80-ms update touches only the newest slot and reuses the cached prefix, avoiding repeated full prefills.
  • The fixed 80-ms grid makes the architecture conversational: lexical output is capped at 12.5 tokens per second, so the same design does not directly serve long-form text generation.
  • Video understanding is not sacrificed by the aligned formulation: the 4B model scores comparably to several 7B online baselines on OVOBench, while being the only evaluated architecture with duplex I/O.
  • Because perception stays active during generation, the model can react to events that occur mid-response, which turn-based streaming models cannot do.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • In our reading, the Section 4.4 ablation suggests the silence-label mask is doing more work than a training detail: at r=0.00 the model is almost mute, so a fair test of the 'timing from one objective' claim would ramp the mask down during training and check whether timing behavior survives.
  • A coarse-to-fine slot structure, with a fast timing scheduler over a slower lexical decoder, could lift the 12.5-token/s ceiling while keeping a shared clock; this is a natural extension the paper lists as adaptive-rate decoding.
  • The aligned-stream format is modular enough that it could be instantiated per speaker to handle multi-party dialogue, with each participant's stream carrying its own silence tokens and the model scheduling turn-taking on the shared grid.
  • A directly testable extension would report response-onset alignment: measure the lag between each initiated response and the event it comments on, rather than only head perplexity or benchmark averages, to show timing quality is learned and not just masked in.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Aero Realtime, a 4B streaming multimodal model that aligns video, audio, and text output on a shared temporal grid of roughly 80-ms slots. Each slot predicts either a lexical token or a special silence token, so input and output advance together under a single autoregressive objective. The authors provide a complete training and serving recipe: causal realtime QA construction, slot-aligned supervision, silence-aware optimization, modality-aware three-level parallelism, and cache-valid delta inference with KV-cache reuse. On OVOBench the 4B model achieves 48.31 average, comparable to several 7B methods though below the strongest baselines. On four A6000 GPUs, the system maintains 84-ms median and 173-ms P95 processing lag over the first 20 minutes of a continuously streamed video. The central claim is that one aligned autoregressive objective can learn both when to respond and what to generate, without an external response gate.

Significance. If the central claim is fully supported, Aero Realtime would be a valuable architectural contribution: it demonstrates a duplex, natively proactive interface that is compatible with KV-cache-reusing inference engines and that retains non-trivial video understanding in a compact 4B model. The paper is unusually complete in its engineering details, including training stages, data mixtures, parallelization strategy, and serving mechanism. The latency measurements over a 20-minute continuous stream are a useful concrete data point, and the authors are candid about limitations such as possible data contamination and the 12.5-token/s output ceiling. However, the paper's headline claim that the aligned objective 'learns when to respond' is not yet demonstrated: the silence-label masking ratio is a hand-tuned hyperparameter that is decisive for whether the model speaks at all, and no direct timing-quality metric is reported. The contribution is therefore promising but requires additional evidence before the central claim can be accepted.

major comments (3)
  1. [Section 4.4, Table 4] The silence-label masking ratio r is load-bearing for the claim in Section 3.2 that 'one autoregressive objective learns both when to respond and what to generate.' With r=0.00 the model 'remains almost entirely silent' and scores only 6.40 on OVOBench, while r=0.95 appears necessary for useful behavior. Since lexical targets are never masked and 95% of [P] targets are dropped from the loss, the effective objective strongly biases the model toward emitting lexical tokens at most slots, so the when-to-respond behavior is substantially imposed by the masking hyperparameter rather than learned by the aligned stream itself. The paper reports no direct timing-quality metric, such as whether the model emits lexical tokens during user speech versus after a question, or how response latency compares with ground-truth response timestamps. I recommend adding an evaluation that measures timing appropriateness directly, and ablating r with respect to that metric, not only OVOBench accuracy.
  2. [Section 4.3, Figure 6, Table 3] The 'sustained realtime processing' claim is only demonstrated for the first 20 minutes of a single 30-minute Video-MME clip. The text reports median and P95 lag over the first 20 minutes and a 153-ms lag at the 20-minute boundary, but gives no information about the 20-to-30-minute interval, where the active sequence grows further. The checkpoint values in Table 3 are non-monotonic (10 min: 95 ms, 15 min: 56 ms, 20 min: 153 ms), so it is unclear whether the system is in a steady state or approaching a threshold. Please report the full 30-minute trajectory, including what happens after 20 minutes, and state whether the KV-cache growth or input backlog is bounded over longer horizons.
  3. [Section 3.2 and Section 4.2] The claim that the model learns 'when to respond' is not directly evaluated anywhere in the experiments. OVOBench measures video understanding under online input, but its Realtime track does not appear to score whether the model's silence/lexical decisions are temporally appropriate. The qualitative examples in Appendix E are illustrative but not quantified. Without a ground-truth-aligned metric for response timing, the paper cannot distinguish 'the model learned when to speak' from 'the masking ratio and the dataset statistics make the model speak at convenient times.' A concrete proposal: use the time-aligned realtime QA data to compute precision/recall of lexical emission within a tolerance window around annotated assistant turn boundaries, and report this for r=0.70 and r=0.95.
minor comments (6)
  1. [Abstract] The abstract states that the system remains 'within 200 ms of the source timeline' but this is only shown for the first 20 minutes of one stream; please qualify the claim as 'for the first 20 minutes of the evaluated stream' to match the evidence.
  2. [Section 4.3] The text says 'a 30-minute Video-MME clip' but reports results only through the first 20 minutes; please clarify whether the remaining 10 minutes were excluded and why.
  3. [Appendix D] The paper reports a single run per configuration without variance; given that the masking-ratio ablation drives the main qualitative change, at least reporting variance across a few seeds for the central configuration would strengthen the results.
  4. [Section 3.2, Eq. (6)] The recurrence in Eq. (6) is attributed to Voxtral Realtime, but the paper does not state what is newly introduced beyond the visual-state injection; a sentence clarifying the architectural delta would help readers position the contribution.
  5. [Section 3.3] The statement that 'path-level deduplication does not guarantee content-level independence from every evaluation video' is honest, but the paper does not quantify the overlap risk; a simple estimate of source-video overlap between the training mixture and OVOBench would be more informative.
  6. [Section 4.5] The claim that 'the larger Backward gain suggests that the focused second-stage mixture particularly improves reasoning over previously observed events' is presented as an interpretation rather than a tested hypothesis; please soften or support it with an additional control.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the aligned-stream objective, training recipe, and latency measurements are self-contained; self-citations are tooling only.

full rationale

I walked the derivation chain from Eq. (9) through the architecture, data conversion, and inference. The central claim—that a single slot-aligned cross-entropy objective over lexical and silence tokens can drive both response timing and content—is supported by the objective being exactly next-token cross-entropy with per-position masking; no quantity used in Eq. (9) is defined in terms of the claimed outcome. The silence-label masking ratio r=0.95 is a hyperparameter, not a fitted parameter renamed as a prediction; the paper reports ablations in Table 4 showing sensitivity, and it does not claim to predict timing from r. The recurrence in Eqs. (6) and (10) follows Voxtral Realtime via citation, but that is external prior work, not a self-citation, and it supplies the recurrence rather than the when-to-respond claim. Self-citations (LMMs-Lab, LMMs-Eval) are training/evaluation tooling and do not carry the central argument. Latency results and OVOBench numbers are external measurements. Thus no circular step can be exhibited by the paper's own equations or by a load-bearing self-citation chain.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced. The silence token [P] is a model vocabulary item, not an invented entity with independent evidence. The central claim rests on one fitted training hyperparameter (silence mask ratio) and several domain assumptions inherited from prior systems and data pipelines.

free parameters (1)
  • Silence-label masking ratio r = 0.95
    Hand-tuned on OVOBench ablations; without masking (r=0) the model never speaks and scores 6.40; with r=0.70 it is imbalanced; r=0.95 selected as best average. This parameter directly controls the when-to-respond behavior.
assumptions (4)
  • domain assumption The 80ms audio slot produced by Qwen2-Audio downsampling is a valid grid for aligning video, audio, and text generation.
    Section 3.2: the model divides continuous time into N audio slots of ~80ms; this is inherited from the audio encoder rather than derived.
  • domain assumption The Voxtral Realtime recurrence (audio plus previous output token) is correct and applicable when extended to multimodal inputs.
    Section 3.2: 'Our causal audio-text recurrence follows the realtime formulation of Voxtral Realtime.'
  • domain assumption GPT-5.4-generated realtime QA pairs are treated as ground truth for when responses should occur.
    Section 3.3: GPT-5.4 generates temporally grounded QA pairs from causal video prefixes; the model is trained to answer at those timestamps.
  • domain assumption OVOBench track scores are a valid proxy for online video understanding of the aligned model.
    Section 4.2: the paper uses OVOBench to claim retained video-understanding capability; possible source overlap is acknowledged as a limitation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation." pith.science (2026). https://pith.science/paper/3UKUOJ6N

@misc{pith2026260808469,
  author       = {Pith},
  title        = {Pith review of: Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3UKUOJ6N}},
  note         = {Machine review of arXiv:2608.08469}
}
read the original abstract

Existing streaming multimodal models process observations incrementally but still follow a turn-based prefill-then-decode pattern, making them non-duplex: new observations cannot naturally enter an active generation stream. Proactive alternatives use micro-turn polling or external response gates, which fragment continuous interaction, decouple response timing from language generation, and complicate KV-cache-friendly serving. We introduce Aero Realtime, a 4B streaming multimodal model with a duplex architecture for realtime generation. Aero Realtime aligns video, audio, and textual output on a shared temporal grid, where each approximately 80-ms audio slot predicts either a lexical token or a silence token. This allows input and output to advance together, enabling one autoregressive objective to learn both when to respond and what to generate. During inference, Aero Realtime appends only the newest multimodal slot, carries forward the previous output state, and reuses the KV cache for efficient incremental execution. We further provide a complete training and serving recipe, including realtime QA construction, slot-aligned supervision, hardware-aware distributed training, and resumable inference. On four NVIDIA A6000 workstation GPUs, Aero Realtime maintains 84-ms median and 173-ms P95 processing lag over 20 minutes of a continuously streamed video, remaining within 200~ms of the source timeline. These results demonstrate the feasibility of fully aligned input-output modeling for duplex, proactive, and hardware-aligned multimodal interaction.

Figures

Figures reproduced from arXiv: 2608.08469 by the authors.

Figure 1
Figure 1. Realtime multimodal architectures: (a) non-duplex turn-based generation, (b) micro-turn polling and external gating, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Aero Realtime architecture. Each audio state is fused with the preceding output embedding; [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Realtime QA construction and training-data composition. Generation uses only past visual context and turns; source [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Throughput and peak memory across packing [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Long-horizon processing lag under continuous [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

109 extracted references · 25 canonical work pages

  1. [1]

    2025 , howpublished=

    Learning to Reason with LLMs , author=. 2025 , howpublished=

    2025

  2. [2]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  3. [3]

    arXiv preprint arXiv:2507.08306 , year=

    M2-reasoning: Empowering mllms with unified general and spatial reasoning , author=. arXiv preprint arXiv:2507.08306 , year=

    arXiv

  4. [4]

    arXiv preprint arXiv:2507.05255 , year=

    Open vision reasoner: Transferring linguistic cognitive behavior for visual reasoning , author=. arXiv preprint arXiv:2507.05255 , year=

    arXiv

  5. [5]

    arXiv preprint arXiv:2509.23661 , year=

    Llava-onevision-1.5: Fully open framework for democratized multimodal training , author=. arXiv preprint arXiv:2509.23661 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2503.07365 , year=

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning , author=. arXiv preprint arXiv:2503.07365 , year=

    arXiv

  7. [7]

    arXiv preprint arXiv:2502.01456 , year=

    Process reinforcement through implicit rewards , author=. arXiv preprint arXiv:2502.01456 , year=

    arXiv

  8. [8]

    2025 , eprint=

    OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles , author=. 2025 , eprint=

    2025

Show all 109 references
  1. [9]

    2025 , eprint=

    SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models , author=. 2025 , eprint=

    2025

  2. [10]

    2025 , eprint=

    Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning , author=. 2025 , eprint=

    2025

  3. [11]

    2025 , eprint=

    SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement , author=. 2025 , eprint=

    2025

  4. [12]

    2025 , eprint=

    LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model , author=. 2025 , eprint=

    2025

  5. [13]

    2025 , eprint=

    LLaVA-Critic: Learning to Evaluate Multimodal Models , author=. 2025 , eprint=

    2025

  6. [14]

    2025 , eprint=

    Qwen2.5-VL Technical Report , author=. 2025 , eprint=

    2025

  7. [15]

    2025 , eprint=

    OpenThoughts: Data Recipes for Reasoning Models , author=. 2025 , eprint=

    2025

  8. [16]

    arXiv preprint arXiv:2410.21276 , year=

    Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year=

    arXiv

  9. [17]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    arXiv

  10. [18]

    2025 , eprint=

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author=. 2025 , eprint=

    2025

  11. [19]

    2024 , eprint=

    MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? , author=. 2024 , eprint=

    2024

  12. [20]

    2024 , eprint=

    We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? , author=. 2024 , eprint=

    2024

  13. [21]

    2024 , eprint=

    LLaVA-OneVision: Easy Visual Task Transfer , author=. 2024 , eprint=

    2024

  14. [22]

    2025 , eprint=

    MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources , author=. 2025 , eprint=

    2025

  15. [23]

    2025 , eprint=

    Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning , author=. 2025 , eprint=

    2025

  16. [24]

    2025 , eprint=

    MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models , author=. 2025 , eprint=

    2025

  17. [25]

    2016 , eprint=

    A Diagram Is Worth A Dozen Images , author=. 2016 , eprint=

    2016

  18. [26]

    2025 , eprint=

    VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning , author=. 2025 , eprint=

    2025

  19. [27]

    2025 , eprint=

    We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning , author=. 2025 , eprint=

    2025

  20. [28]

    Aniruddha Kembhavi and Min Joon Seo and Dustin Schwenk and Jonghyun Choi and Ali Farhadi and Hannaneh Hajishirzi , title =. 2017. 2017 , url =. doi:10.1109/CVPR.2017.571 , timestamp =

    2017 doi

  21. [29]

    2024 , eprint=

    Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoning , author=. 2024 , eprint=

    2024

  22. [30]

    2024 , eprint=

    PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns , author=. 2024 , eprint=

    2024

  23. [31]

    Open R1: A fully open reproduction of DeepSeek-R1 , url =

  24. [32]

    2024 , month = dec, day =

    Kavukcuoglu, Koray and Hassabis, Demis and Pichai, Sundar , title =. 2024 , month = dec, day =

    2024

  25. [33]

    2024 , eprint=

    GPT-4o System Card , author=. 2024 , eprint=

    2024

  26. [34]

    2025 , eprint=

    InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models , author=. 2025 , eprint=

    2025

  27. [35]

    2024 , eprint=

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint=

    2024

  28. [36]

    2025 , eprint=

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale , author=. 2025 , eprint=

    2025

  29. [37]

    2025 , eprint=

    Group Sequence Policy Optimization , author=. 2025 , eprint=

    2025

  30. [38]

    2024 , eprint=

    MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI , author=. 2024 , eprint=

    2024

  31. [39]

    2024 , eprint=

    Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset , author=. 2024 , eprint=

    2024

  32. [40]

    , author=

    LMMs Engine: A simple, unified multimodal framework for pretraining and finetuning. , author=. 2025 , url=

    2025

  33. [41]

    2024 , eprint=

    LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models , author=. 2024 , eprint=

    2024

  34. [42]

    LMMs-Eval: Accelerating the Development of Large Multimoal Models , url=

    Bo Li* and Peiyuan Zhang* and Kaichen Zhang* and Fanyi Pu* and Xinrun Du and Yuhao Dong and Haotian Liu and Yuanhan Zhang and Ge Zhang and Chunyuan Li and Ziwei Liu , publisher =. LMMs-Eval: Accelerating the Development of Large Multimoal Models , url=

  35. [43]

    Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

  36. [44]

    2024 , journal =

    HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal =

    2024

  37. [45]

    Liger-Kernel: Efficient Triton Kernels for

    Pin-Lun Hsu and Yun Dai and Vignesh Kothapalli and Qingquan Song and Shao Tang and Siyu Zhu and Steven Shimizu and Shivam Sahni and Haowen Ning and Yanning Chen and Zhipeng Wang , booktitle=. Liger-Kernel: Efficient Triton Kernels for. 2025 , url=

    2025

  38. [46]

    2025 , eprint=

    LLaVA-CoT: Let Vision Language Models Reason Step-by-Step , author=. 2025 , eprint=

    2025

  39. [47]

    2025 , eprint=

    R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization , author=. 2025 , eprint=

    2025

  40. [48]

    2025 , eprint=

    Kimi k1.5: Scaling Reinforcement Learning with LLMs , author=. 2025 , eprint=

    2025

  41. [49]

    2025 , eprint=

    LIMO: Less is More for Reasoning , author=. 2025 , eprint=

    2025

  42. [50]

    2025 , eprint=

    MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization , author=. 2025 , eprint=

    2025

  43. [51]

    2025 , eprint=

    MiMo-VL Technical Report , author=. 2025 , eprint=

    2025

  44. [52]

    2024 , eprint=

    MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts , author=. 2024 , eprint=

    2024

  45. [53]

    2024 , eprint=

    LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts , author=. 2024 , eprint=

    2024

  46. [54]

    2025 , eprint=

    DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models , author=. 2025 , eprint=

    2025

  47. [55]

    2025 , eprint=

    MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark , author=. 2025 , eprint=

    2025

  48. [56]

    2024 , eprint=

    CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs , author=. 2024 , eprint=

    2024

  49. [57]

    2025 , note =

    Qwen3‑VL: Sharper Vision, Deeper Thought, Broader Action , howpublished =. 2025 , note =

    2025

  50. [58]

    2025 , eprint=

    Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning , author=. 2025 , eprint=

    2025

  51. [59]

    2025 , eprint=

    Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning , author=. 2025 , eprint=

    2025

  52. [60]

    2025 , eprint=

    VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning , author=. 2025 , eprint=

    2025

  53. [61]

    Thinking with Long Videos

    LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling , author=. arXiv preprint arXiv:2511.20785 , year=

    arXiv

  54. [62]

    2023 , eprint=

    Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding , author=. 2023 , eprint=

    2023

  55. [63]

    2024 , eprint=

    VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs , author=. 2024 , eprint=

    2024

  56. [64]

    2025 , eprint=

    VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding , author=. 2025 , eprint=

    2025

  57. [65]

    2026 , note=

    Marin: An Open Lab for Building Foundation Models Together , author=. 2026 , note=

    2026

  58. [66]

    CVPR , year=

    Chen, Joya and Lv, Zhaoyang and Wu, Shiwei and Lin, Kevin Qinghong and Song, Chenan and Gao, Difei and Liu, Jia-Wei and Gao, Ziteng and Mao, Dongxing and Shou, Mike Zheng , title=. CVPR , year=. 2406.11816 , archivePrefix=

    arXiv

  59. [67]

    ICCV , year=

    Zhang, Haoji and Wang, Yiqin and Tang, Yansong and Liu, Yong and Feng, Jiashi and Jin, Xiaojie , title=. ICCV , year=. 2506.23825 , archivePrefix=

    arXiv

  60. [68]

    CVPR , year=

    Qian, Rui and Ding, Shuangrui and Dong, Xiaoyi and Zhang, Pan and Zang, Yuhang and Cao, Yuhang and Lin, Dahua and Wang, Jiaqi , title=. CVPR , year=. 2501.03218 , archivePrefix=

    arXiv

  61. [69]

    ACM Multimedia , year=

    Yao, Linli and Li, Yicheng and Wei, Yuancheng and Li, Lei and Ren, Shuhuai and Liu, Yuanxin and Ouyang, Kun and Wang, Lean and Li, Shicheng and Li, Sida and Kong, Lingpeng and Liu, Qi and Zhang, Yuanxing and Sun, Xu , title=. ACM Multimedia , year=. 2504.17343 , archivePrefix=

    arXiv

  62. [70]

    arXiv preprint arXiv:2509.24871 , year=

    Zeng, Xiangyu and Qiu, Kefan and Zhang, Qingyu and Li, Xinhao and Wang, Jing and Li, Jiaxin and Yan, Ziang and Tian, Kun and Tian, Meng and Zhao, Xinhai and Wang, Yi and Wang, Limin , title=. arXiv preprint arXiv:2509.24871 , year=

  63. [71]

    arXiv preprint arXiv:2512.21334 , year=

    Xia, Jiaer and Chen, Peixian and Zhang, Mengdan and Sun, Xing and Zhou, Kaiyang , title=. arXiv preprint arXiv:2512.21334 , year=

    arXiv

  64. [72]

    arXiv preprint arXiv:2601.14724 , year=

    Zhang, Haowei and Yang, Shudong and Fu, Jinlan and Ng, See-Kiong and Qiu, Xipeng , title=. arXiv preprint arXiv:2601.14724 , year=

    arXiv

  65. [73]

    CVPR , year=

    Chen, Joya and Zeng, Ziyun and Lin, Yiqi and Li, Wei and Ma, Zejun and Shou, Mike Zheng , title=. CVPR , year=. 2504.16030 , archivePrefix=

    arXiv

  66. [74]

    CVPR , year=

    Wang, Yuxuan and Wang, Yueqian and Chen, Bo and Wu, Tong and Zhao, Dongyan and Zheng, Zilong , title=. CVPR , year=. 2503.22952 , archivePrefix=

    arXiv

  67. [75]

    NeurIPS , year=

    Wang, Haibo and Feng, Bo and Lai, Zhengfeng and Xu, Mingze and Li, Shiyu and Ge, Weifeng and Dehghan, Afshin and Cao, Meng and Huang, Ping , title=. NeurIPS , year=. 2505.05467 , archivePrefix=

  68. [76]

    Findings of EMNLP , year=

    Wang, Yueqian and Meng, Xiaojun and Wang, Yuxuan and Liang, Jianxin and Wei, Jiansheng and Zhang, Huishuai and Zhao, Dongyan , title=. Findings of EMNLP , year=. 2411.17991 , archivePrefix=

  69. [77]

    ICLR , year=

    Xu, Ruyi and Xiao, Guangxuan and Chen, Yukang and He, Liuning and Peng, Kelly and Lu, Yao and Han, Song , title=. ICLR , year=. 2510.09608 , archivePrefix=

    arXiv

  70. [78]

    arXiv preprint arXiv:2603.08620 , year=

    Azad, Shehreen and Vineet, Vibhav and Rawat, Yogesh Singh , title=. arXiv preprint arXiv:2603.08620 , year=

  71. [79]

    arXiv preprint arXiv:2603.12262 , year=

    Guan, Yiran and Yin, Liang and Liang, Dingkang and Ju, Jianzhong and Luo, Zhenbo and Luan, Jian and Liu, Yuliang and Bai, Xiang , title=. arXiv preprint arXiv:2603.12262 , year=

    arXiv

  72. [80]

    arXiv preprint arXiv:2601.15655 , year=

    Guo, Zhenghui and Man, Yuanbin and Sheng, Junyuan and Lin, Bowen and Ahmed, Ahmed and Jiang, Bo and Zhang, Boyuan and Yin, Miao and Jin, Sian and Gnawal, Omprakash and Zhang, Chengming , title=. arXiv preprint arXiv:2601.15655 , year=

  73. [81]

    arXiv preprint arXiv:2603.03985 , year=

    Shi, Yansong and Zhao, Qingsong and Jiang, Tianxiang and Zeng, Xiangyu and Wang, Yi and Wang, Limin , title=. arXiv preprint arXiv:2603.03985 , year=

  74. [82]

    arXiv preprint arXiv:2603.02872 , year=

    Zhang, Jialiang and Tong, Junlong and Lin, Junyan and Wu, Hao and Sun, Yirong and Ma, Yunpu and Shen, Xiaoyu , title=. arXiv preprint arXiv:2603.02872 , year=

  75. [83]

    CVPR , year=

    Zhou, Xingyi and Arnab, Anurag and Buch, Shyamal and Yan, Shen and Myers, Austin and Xiong, Xuehan and Nagrani, Arsha and Schmid, Cordelia , title=. CVPR , year=. 2404.01297 , archivePrefix=

    arXiv

  76. [84]

    CVPR , year=

    Li, Yifei and Niu, Junbo and Miao, Ziyang and Ge, Chunjiang and Zhou, Yuanhang and He, Qihao and Dong, Xiaoyi and Duan, Haodong and Ding, Shuangrui and Qian, Rui and Zhang, Pan and Zang, Yuhang and Cao, Yuhang and He, Conghui and Wang, Jiaqi , title=. CVPR , year=. 2501.05510 ...

    arXiv

  77. [85]

    arXiv preprint arXiv:2404.16821 , year=

    Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and Ma, Ji and Wang, Jiaqi and Dong, Xiaoyi and Yan, Hang and Guo, Hewei and He, Conghui and Shi, Botian and Jin, Zhenjia...

    arXiv

  78. [86]

    arXiv preprint arXiv:2409.12191 , year=

    Wang, Peng and Bai, Shuai and Tan, Sinan and Wang, Shijie and Fan, Zhihao and Bai, Jinze and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and others , title=. arXiv preprint arXiv:2409.12191 , year=

    arXiv

  79. [87]

    Transactions on Machine Learning Research , year=

    Zhang, Yuanhan and Li, Bo and Liu, Haotian and Lee, Yong Jae and Gui, Liangke and Fu, Di and Feng, Jiashi and Liu, Ziwei and Li, Chunyuan , title=. Transactions on Machine Learning Research , year=. 2410.02713 , archivePrefix=

    arXiv

  80. [88]

    ICML , year=

    Shen, Xiaoqian and Xiong, Yunyang and Zhao, Changsheng and Wu, Lemeng and Chen, Jun and Zhu, Chenchen and Liu, Zechun and Xiao, Fanyi and Varadarajan, Balakrishnan and Borber, Florian and others , title=. ICML , year=. 2410.17434 , archivePrefix=

    arXiv

  81. [89]

    2026 , eprint=

    A Simple Baseline for Streaming Video Understanding , author=. 2026 , eprint=

    2026

  82. [90]

    2026 , eprint=

    VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding , author=. 2026 , eprint=

    2026

  83. [91]

    2026 , month=

    Interaction Models: A Scalable Approach to Human-AI Collaboration , journal=. 2026 , month=

    2026

  84. [92]

    2025 , eprint=

    StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition , author=. 2025 , eprint=

    2025

  85. [93]

    arXiv preprint arXiv:2602.02204 , year=

    vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models , author=. arXiv preprint arXiv:2602.02204 , year=

  86. [94]

    2605.25979 , archivePrefix=

    Xiang An and Yin Xie and Feilong Tang and Yunyao Yan and Huajie Tan and Didi Zhu and Changrui Chen and Xiuwei Zhao and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Kaichen Zhang and Wenkang Zhang and Zheng Cheng and Nansen Zhang and Chunsheng Wu and Chunjiang...

    arXiv

  87. [95]

    2025 , eprint=

    Qwen2.5-Omni Technical Report , author=. 2025 , eprint=

    2025

  88. [96]

    2025 , eprint=

    Qwen3-Omni Technical Report , author=. 2025 , eprint=

    2025

  89. [97]

    2026 , eprint=

    EgoLife: Towards Egocentric Life Assistant , author=. 2026 , eprint=

    2026

  90. [98]

    2025 , eprint=

    Advancing Egocentric Video Question Answering with Multimodal Large Language Models , author=. 2025 , eprint=

    2025

  91. [99]

    2025 , eprint=

    Qwen3-VL Technical Report , author=. 2025 , eprint=

    2025

  92. [100]

    2024 , eprint=

    Qwen2-Audio Technical Report , author=. 2024 , eprint=

    2024

  93. [101]

    2026 , eprint=

    AURA: Always-On Understanding and Real-Time Assistance via Video Streams , author=. 2026 , eprint=

    2026

  94. [102]

    2026 , eprint=

    JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence , author=. 2026 , eprint=

    2026

  95. [103]

    2025 , eprint=

    Can Vision-Language Models Answer Face to Face Questions in the Real-World? , author=. 2025 , eprint=

    2025

  96. [104]

    2026 , eprint=

    Voxtral Realtime , author=. 2026 , eprint=

    2026

  97. [105]

    2026 , eprint=

    MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention , author=. 2026 , eprint=

    2026

  98. [106]

    2024 , eprint=

    Moshi: A Speech-Text Foundation Model for Real-Time Dialogue , author=. 2024 , eprint=

    2024

  99. [107]

    2025 , eprint=

    Vision-Speech Models: Teaching Speech Models to Converse about Images , author=. 2025 , eprint=

    2025

  100. [108]

    2026 , eprint=

    Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model , author=. 2026 , eprint=

    2026

  101. [109]

    2023 , eprint=

    DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models , author=. 2023 , eprint=

    2023

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.