Pith. sign in

REVIEW 3 major objections 5 minor 42 references

Query-guided, modality-symmetric token compression lets omni-modal models keep most audio-visual accuracy while dropping most tokens, without letting one modality steer the cut.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 05:12 UTC pith:5DOQJF4H

load-bearing objection Solid training-free fix for audio-guided modality bias in omni token compression; real but incremental systems result, with budget calibration as the main fairness soft spot. the 3 major comments →

arxiv 2607.03050 v1 pith:5DOQJF4H submitted 2026-07-03 cs.LG cs.AIcs.CVcs.SD

OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models

Shijie Cao , Qingyu Zhang , Boxi Yu , Yuzhong Zhang , Boxi Cao , Yaojie Lu , Hongyu Lin , Xianpei Han
show 1 more author
Le Sun
This is my paper
classification cs.LG cs.AIcs.CVcs.SD
keywords token compressionomni-modal large language modelsaudio-visual understandingquery-guided compressionmodality biastraining-free inferencetemporal chunk budgeting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Omni-modal language models that read video and audio together produce huge token sequences, so inference is expensive. Prior compression often lets one modality (often audio) decide which time chunks to keep, which the authors argue both ignores where the query’s evidence actually sits and assumes the two streams share the same information density over time. OmniFocus instead scores each temporal chunk against the text query separately for video and for audio, turns those scores into local drop budgets under fixed global retention, then keeps tokens that either align across modalities or stand out inside their own modality. On the Qwen2.5-Omni 3B and 7B models across four audio-visual benchmarks, this training-free recipe holds accuracy better than random, vision-style, and audio-guided baselines at 35% and 25% retention, including 59.40 accuracy on DailyOmni at 25% retention with up to 1.38× prefill speedup on the 7B model. A sympathetic reader cares because long-form joint audio-visual understanding only becomes practical if compression can cut tokens without systematically discarding the modality the question actually needs.

Core claim

The paper claims that unimodal-guided audio-visual token compression induces modality bias, and that a training-free alternative—independent query-token relevance per modality, chunk-wise budget allocation, and hybrid inter-modal plus intra-modal token selection—preserves both cross-modal alignment and modality-specific evidence well enough to outperform existing baselines on several major scores at 25% retention while delivering measurable prefill and end-to-end speedups.

What carries the argument

OmniFocus: before the language backbone, mean-normalized query embeddings score each audio and video chunk by max cosine similarity; those scores become calibrated local drop ratios under global modality budgets; within each chunk, retained tokens are split between high inter-modal association and high intra-modal peak scores.

Load-bearing premise

Cosine similarity in the frozen input embedding space between the query and audio or video tokens is a reliable enough importance signal to decide which chunks and tokens to keep without using model attention or task training.

What would settle it

On DailyOmni AV Event Alignment at 25% retention, if independent query-max scoring is replaced by pure audio-side guidance (or by noise embeddings) and the visual/joint accuracy gap closes or reverses while overall average still matches OmniFocus, the claim that query-guided modality-symmetric scoring is what mitigates modality bias would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • OmniLLM inference on long daily videos can target ~25% token retention with only modest average accuracy loss when budgets are query- and modality-aware.
  • Audio-guided temporal budgeting alone is an unreliable default when questions need visual or joint evidence.
  • Preserving both cross-modal correspondence and within-modality peaks is more stable than either cue alone under tight budgets.
  • Relative memory and latency gains grow with video duration, so longer omnimodal inputs benefit most from this style of pre-backbone compression.
  • Training-free plug-in compressors remain viable for existing OmniLLMs without retraining the backbone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If frozen-embedding query match is the bottleneck, light calibration that mixes early-layer attention with the current scores could lift fine-grained event questions without full finetuning.
  • Instance-adaptive audio/video global budgets (instead of benchmark-level calibration) would test whether modality bias is mainly a budgeting problem or a selection problem.
  • The same independent-modality query scoring pattern may transfer to other interleaved multimodal streams (e.g., multi-camera plus audio) where one stream currently sets the prune schedule.
  • Failure modes under noisy audio, occlusion, or deliberate A/V desync would show whether inter-modal association tokens become harmful rather than helpful.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes OmniFocus, a training-free token compression method for OmniLLMs that estimates query relevance independently for audio and video chunks via max cosine similarity in the frozen input embedding space, maps those scores to chunk-wise local drop ratios under calibrated global modality budgets, and retains tokens with a hybrid of inter-modal association and intra-modal peak scores. It argues that unimodal-guided compressors (notably OmniZip) introduce modality bias by assuming temporally aligned information density across modalities and by ignoring query-local evidence. Experiments on Qwen2.5-Omni-3B/7B across DailyOmni, WorldSense, OmniVideoBench, and VideoMME at 35% and 25% retention report competitive or best compressed accuracy versus Random, DyCoke, and OmniZip, with efficiency gains (e.g., DailyOmni 59.40 accuracy and 1.38× prefill speedup at 25% on 7B). Ablations support max-similarity scoring, score-based allocation, hybrid inter+intra selection, and balanced modality scoring.

Significance. If the comparative gains hold under stricter fairness controls, the work is a useful systems contribution for long audio-visual OmniLLM inference: a simple, training-free, modality-symmetric alternative to audio-guided budgeting that is easy to plug in before the LLM backbone. Strengths include a clear problem framing of modality bias, a fully specified three-stage pipeline, multi-benchmark evaluation, efficiency profiling, and reasonably thorough ablations (Tables 4–5, Figures 3–6, Appendix D). The contribution is incremental rather than foundational—gains are often small and model-family-specific—but the accuracy–efficiency trade-off is practically relevant for long-form omni-modal serving.

major comments (3)
  1. Appendix A and §4.1 state that global audio/video drop ratios are calibrated per benchmark after following the OmniZip protocol “to ensure fair comparison.” This is load-bearing for the central comparative claim (Tables 1–3), yet the paper does not report the exact A/V budgets used, nor a fixed shared-budget control where all methods receive identical global modality retention without post-hoc adjustment. Without that control, it is hard to separate query-symmetric design from budget tuning, especially where average gains over OmniZip are ~0.2–2 points.
  2. All main results are single-run scores on only Qwen2.5-Omni-3B/7B (Tables 1–3; checklist item 7). Given small margins on WorldSense/OmniVideoBench/VideoMME averages and the free parameters listed in Appendix A/D (rmin/rmax, inter/intra split, modality budgets), multi-seed or multi-run variance—or at least one additional OmniLLM family—is needed to support “outperforms existing baselines on several major benchmark scores” as a robust claim rather than a family-specific observation.
  3. §3.2 Eqs. (2)–(3) allocate chunk budgets from frozen embedding max-similarity alone, without LLM attention or task supervision. Limitations/Appendix E acknowledge this, but the paper lacks a direct failure-mode analysis (e.g., ambiguous queries, weak embedding alignment, or cases where max-similarity peaks on non-causal tokens). A small diagnostic set or qualitative retention examples would better ground the weakest methodological assumption behind the reported gains.
minor comments (5)
  1. Figure 1’s modality-type protocol is deferred to Appendix B; a one-sentence summary in the main text would make the figure self-contained.
  2. Table 3 OmniZip DailyOmni accuracies (58.98/56.98 on 7B) do not exactly match Table 1 averages (59.23/57.73); reconcile or footnote the difference.
  3. Notation for chunk token counts (ni_v / ni_a) and keep counts ki_m is clear, but the clamp/binary-search calibration in Eq. (6) would benefit from a short pseudocode box.
  4. Checklist items note missing licenses for assets, no broader-impacts discussion, and incomplete total-compute reporting; these should be fixed for camera-ready.
  5. Code is cited via a GitHub URL but the checklist marks open access as No; either release anonymized code or remove the implication of availability at submission.

Circularity Check

0 steps flagged

No circular derivation: OmniFocus is an empirical, training-free compression design tested on external benchmarks, not a prediction forced by its inputs.

full rationale

This paper proposes a training-free engineering method (query-token cosine similarity for chunk importance, calibrated local drop ratios, inter+intra token selection) and evaluates it empirically against OmniZip, DyCoke, and random retention on public audio-visual benchmarks with Qwen2.5-Omni. There is no first-principles derivation chain in which a claimed prediction reduces to a fitted parameter or to a definition of the same quantity. Global A/V budget calibration (Appendix A) is an evaluation-protocol choice for comparable retention across methods, not a fit that is then re-labeled as a prediction; the method’s scoring and selection rules remain independently specified and are not true by construction of those budgets. Related-work citations (OmniZip, DyCoke, etc.) are external baselines, not load-bearing self-citation uniqueness theorems. Ablations vary design choices and report measured accuracy, which is ordinary empirical science rather than circular reasoning. Per the analyzer’s default, this is a self-contained empirical methods paper with no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central empirical claim rests on a small set of hand-chosen compression hyperparameters, the assumption that frozen embedding similarity proxies query relevance, and the evaluation protocol that calibrates modality budgets per benchmark. No new physical entities are postulated; the invented objects are algorithmic constructs.

free parameters (4)
  • local drop-ratio range [rmin, rmax]
    Fixed to 0.35–0.75 in all main experiments; Appendix D.6 shows performance moves when the range changes.
  • global audio/video retention budgets
    Calibrated per benchmark following OmniZip protocol then adjusted for fair comparison; not instance-adaptive.
  • inter/intra keep-count split
    Default equal 1:1 split is a parameter-free choice; Appendix D.3 shows sensitivity to other splits.
  • temporal window size
    Uses the model default 2-second window for chunking; compression decisions inherit this granularity.
axioms (4)
  • domain assumption Query-relevant evidence is temporally sparse and can be captured by max cosine similarity in the frozen LLM input embedding space.
    Core of §3.2 scoring; Limitations note this may fail for fine-grained or ambiguous queries.
  • ad hoc to paper Audio and video should receive independent importance estimates rather than one modality guiding the other.
    Design premise motivated by Figure 1 and modality-bias diagnosis; validated mainly by ablations on DailyOmni.
  • ad hoc to paper Inter-modal association plus intra-modal peak scores are sufficient complementary criteria for token retention under a fixed budget.
    Dual-score selection in §3.2; hybrid beats either alone in Figure 3.
  • domain assumption Benchmark-level modality budget calibration yields fair comparisons across compression methods.
    Appendix A protocol used for all methods; still a non-trivial evaluation assumption.
invented entities (2)
  • OmniFocus compression pipeline no independent evidence
    purpose: Training-free query-guided, modality-symmetric token compressor for OmniLLMs.
    The paper’s main algorithmic object; evidence is empirical benchmark comparison, not independent external measurement.
  • Dual inter-modal association / intra-modal peak token scores no independent evidence
    purpose: Select retained tokens that preserve cross-modal alignment and modality-specific saliency.
    Defined in §3.2 equations (7)–(9); ablations support usefulness but no external theory validates the equal split.

pith-pipeline@v1.1.0-grok45 · 26460 in / 3143 out tokens · 23798 ms · 2026-07-12T05:12:21.208725+00:00 · methodology

0 comments
read the original abstract

Omni modal large language models (OmniLLMs) have attracted wide attention for their ability to jointly process audio and video, but they generate large token sequences under audio-visual inputs, leading to substantial inference cost. Existing audio-visual token compression methods often rely on unimodal guidance, overlooking the temporal locality of query-relevant evidence in audio-visual inputs and implicitly assuming that the two modalities share a temporally aligned information density distribution. We propose \textbf{OmniFocus}, a training-free query-guided token compression method for OmniLLMs that performs independent importance estimation for video and audio, enabling a modality-symmetric compression design that preserves modality-specific salient evidence while maintaining audio-visual alignment, thereby mitigating the modality bias issue that can arise from unimodal-guided compression. Experiments on the Qwen2.5-Omni model family across four audio-visual benchmarks show that OmniFocus maintains strong compressed performance at low token retention ratios and outperforms existing baselines on several major benchmark scores at 25\% token retention. On DailyOmni with Qwen2.5-Omni-7B at 25\% token retention, OmniFocus maintains 59.40 accuracy while delivering up to 1.38$\times$ prefill speedup relative to the full-token baseline, highlighting a favorable practical accuracy-efficiency trade-off.

Figures

Figures reproduced from arXiv: 2607.03050 by Boxi Cao, Boxi Yu, Hongyu Lin, Le Sun, Qingyu Zhang, Shijie Cao, Xianpei Han, Yaojie Lu, Yuzhong Zhang.

Figure 1
Figure 1. Figure 1: Illustration of the limitation of unimodal-guided audio-visual token compression. Audio [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of OmniFocus. Given a textual query and temporally aligned audio-video chunks, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Impact of inter- and intra-modal to￾ken selection. We compare inter-modal only, intra￾modal only, and the hybrid inter+intra strategy un￾der 35% token retention for 3B and 7B models [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Efficiency gains by video duration on WorldSense. We group QA examples into 100- second video-duration buckets and report GPU memory reduction and end-to-end time reduction relative to the same-model full-token baseline. Longer videos generally yield larger relative savings, and the stronger compression setting (25%) provides the largest gains. at 25% retention, while the corresponding end-to-end time redu… view at source ↗
Figure 5
Figure 5. Figure 5: Sensitivity to modality-specific retention ratios. We vary the audio retention ratio while fixing the video ratio (left), and vary the video retention ratio while fixing the audio ratio (right). OmniFocus is robust around moderate modality budgets, while overly aggressive compression of either modality degrades DailyOmni performance [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Sensitivity to the inter/intra token-selection split on DailyOmni for Qwen2.5-Omni-7B. We report average accuracy under different keep-count splits between inter-modal association scores and intra-modal peak scores at 35% and 25% token retention. The default 1:1 split is highlighted, and the best point in each retention setting is marked [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 16 linked inside Pith

  1. [1]

    Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. InEuropean Conference on Computer Vision, pages 19–35. Springer, 2024

    2024

  3. [3]

    Event-anchored frame selection for effective long-video understanding.arXiv preprint arXiv:2603.00983, 2026

    Wang Chen, Yongdong Luo, Yuhui Zeng, Luojun Lin, Tianyu Xie, Fei Chao, Rongrong Ji, and Xiawu Zheng. Event-anchored frame selection for effective long-video understanding.arXiv preprint arXiv:2603.00983, 2026

    arXiv 2026

  4. [4]

    Avocado: An audiovisual video captioner driven by temporal orchestration.arXiv preprint arXiv:2510.10395, 2025

    Xinlong Chen, Yue Ding, Weihong Lin, Jingyun Hua, Linli Yao, Yang Shi, Bozhou Li, Yuanxing Zhang, Qiang Liu, Pengfei Wan, et al. Avocado: An audiovisual video captioner driven by temporal orchestration.arXiv preprint arXiv:2510.10395, 2025

    arXiv 2025

  5. [5]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Streaming video question-answering with in-context video kv-cache retrieval.arXiv preprint arXiv:2503.00540, 2025

    Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. Streaming video question-answering with in-context video kv-cache retrieval.arXiv preprint arXiv:2503.00540, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Omnisift: Modality-asymmetric token compression for efficient omni-modal large language models.arXiv preprint arXiv:2602.04804, 2026

    Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, et al. Omnisift: Modality-asymmetric token compression for efficient omni-modal large language models.arXiv preprint arXiv:2602.04804, 2026

    Pith/arXiv arXiv 2026

  8. [8]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24108–24118, 2025

    2025

  9. [9]

    Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos

    Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18959–18969, 2025

    2025

  10. [10]

    Mmou: A massive multi-task omni understanding and reasoning benchmark for long and complex real-world videos.arXiv preprint arXiv:2603.14145, 2026

    Arushi Goel, Sreyan Ghosh, Vatsal Agarwal, Nishit Anand, Kaousheik Jayakumar, Lasha Koroshinadze, Yao Xu, Katie Lyons, James Case, Karan Sapra, et al. Mmou: A massive multi-task omni understanding and reasoning benchmark for long and complex real-world videos.arXiv preprint arXiv:2603.14145, 2026

    Pith/arXiv arXiv 2026

  11. [11]

    Echoing- pixels: Cross-modal adaptive token reduction for efficient audio-visual llms.arXiv preprint arXiv:2512.10324, 2025

    Chao Gong, Depeng Wang, Zhipeng Wei, Ya Guo, Huijia Zhu, and Jingjing Chen. Echoing- pixels: Cross-modal adaptive token reduction for efficient audio-visual llms.arXiv preprint arXiv:2512.10324, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Thinkomni: Lifting textual reasoning to omni-modal scenarios via guidance decoding.arXiv preprint arXiv:2602.23306, 2026

    Yiran Guan, Sifan Tu, Dingkang Liang, Linghao Zhu, Jianzhong Ju, Zhenbo Luo, Jian Luan, Yuliang Liu, and Xiang Bai. Thinkomni: Lifting textual reasoning to omni-modal scenarios via guidance decoding.arXiv preprint arXiv:2602.23306, 2026

    arXiv 2026

  13. [13]

    Zipvl: Efficient large vision-language models with dynamic token sparsification.arXiv preprint arXiv:2410.08584, 2024

    Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipvl: Efficient large vision-language models with dynamic token sparsification.arXiv preprint arXiv:2410.08584, 2024. 10

    Pith/arXiv arXiv 2024

  14. [14]

    Worldsense: Evaluat- ing real-world omnimodal understanding for multimodal llms.arXiv preprint arXiv:2502.04326, 2025

    Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluat- ing real-world omnimodal understanding for multimodal llms.arXiv preprint arXiv:2502.04326, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Prunevid: Visual token pruning for efficient video large language models

    Xiaohu Huang, Hao Zhou, and Kai Han. Prunevid: Visual token pruning for efficient video large language models. InFindings of the Association for Computational Linguistics: ACL 2025, pages 19959–19973, 2025

    2025

  16. [16]

    From specific-mllms to omni-mllms: a survey on mllms aligned with multi-modalities

    Shixin Jiang, Jiafeng Liang, Jiyuan Wang, Xuan Dong, Heng Chang, Weijiang Yu, Jinhua Du, Ming Liu, and Bing Qin. From specific-mllms to omni-mllms: a survey on mllms aligned with multi-modalities. InFindings of the Association for Computational Linguistics: ACL 2025, pages 8617–8652, 2025

    2025

  17. [17]

    Token pruning in audio transformers: Optimizing performance and decoding patch importance.arXiv preprint arXiv:2504.01690, 2025

    Taehan Lee and Hyukjun Lee. Token pruning in audio transformers: Optimizing performance and decoding patch importance.arXiv preprint arXiv:2504.01690, 2025

    arXiv 2025

  18. [18]

    Dash: Dynamic audio-driven semantic chunking for efficient omnimodal token compression.arXiv preprint arXiv:2603.15685, 2026

    Bingzhou Li and Tao Huang. Dash: Dynamic audio-driven semantic chunking for efficient omnimodal token compression.arXiv preprint arXiv:2603.15685, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Omnivideobench: Towards audio-visual understanding evaluation for omni mllms.arXiv preprint arXiv:2510.10689, 2025

    Caorui Li, Yu Chen, Yiyan Ji, Jin Xu, Zhenyu Cui, Shihao Li, Yuanxing Zhang, Wentao Wang, Zhenghao Song, Dingling Zhang, et al. Omnivideobench: Towards audio-visual understanding evaluation for omni mllms.arXiv preprint arXiv:2510.10689, 2025

    arXiv 2025

  20. [20]

    Omnigaia: Towards native omni-modal ai agents

    Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Shijian Wang, Guanting Dong, Jiajie Jin, Hao Wang, Yinuo Wang, Ji-Rong Wen, Yuan Lu, et al. Omnigaia: Towards native omni-modal ai agents. arXiv preprint arXiv:2602.22897, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    Baichuan-omni-1.5 technical report.arXiv preprint arXiv:2501.15368, 2025

    Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. Baichuan-omni-1.5 technical report.arXiv preprint arXiv:2501.15368, 2025

    arXiv 2025

  22. [22]

    Speechprune: Context-aware token pruning for speech information retrieval

    Yueqian Lin, Yuzhe Fu, Jingyang Zhang, Yudong Liu, Jianyi Zhang, Jingwei Sun, Hai Helen Li, and Yiran Chen. Speechprune: Context-aware token pruning for speech information retrieval. In2025 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2025

    2025

  23. [23]

    Humansense: From multimodal perception to empathetic context-aware responses through reasoning mllms

    Zheng Qin, Ruobing Zheng, Yabing Wang, Tianqi Li, Yi Yuan, Jingdong Chen, and Le Wang. Humansense: From multimodal perception to empathetic context-aware responses through reasoning mllms. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 24973–24981, 2026

    2026

  24. [24]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 22857–22867, 2025

    2025

  25. [25]

    Holitom: Holistic token merging for fast video large language models.arXiv preprint arXiv:2505.21334, 2025

    Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Holitom: Holistic token merging for fast video large language models.arXiv preprint arXiv:2505.21334, 2025

    arXiv 2025

  26. [26]

    When tokens talk too much: A survey of multimodal long- context token compression across images, videos, and audios.arXiv preprint arXiv:2507.20198, 2025

    Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, and Huan Wang. When tokens talk too much: A survey of multimodal long- context token compression across images, videos, and audios.arXiv preprint arXiv:2507.20198, 2025

    arXiv 2025

  27. [27]

    Adaptive keyframe sampling for long video understanding

    Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 29118–29128, 2025

    2025

  28. [28]

    Dycoke: Dynamic compression of tokens for fast video large language models

    Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18992–19001, 2025

    2025

  29. [29]

    Omnizip: Audio- guided dynamic token compression for fast omnimodal large language models.arXiv preprint arXiv:2511.14582, 2025

    Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Huan Wang, et al. Omnizip: Audio- guided dynamic token compression for fast omnimodal large language models.arXiv preprint arXiv:2511.14582, 2025. 11

    Pith/arXiv arXiv 2025

  30. [30]

    Qwen Team. Qwen3. 5-omni technical report.arXiv preprint arXiv:2604.15804, 2026

    Pith/arXiv arXiv 2026

  31. [31]

    Emoomni: Bridging emotional understanding and expression in omni-modal llms.arXiv preprint arXiv:2602.21900, 2026

    Wenjie Tian, Zhixian Zhao, Jingbin Hu, Huakang Chen, Haohe Liu, Binshen Mu, and Lei Xie. Emoomni: Bridging emotional understanding and expression in omni-modal llms.arXiv preprint arXiv:2602.21900, 2026

    arXiv 2026

  32. [32]

    Interactiveomni: A unified omni-modal model for audio-visual multi-turn dialogue.arXiv preprint arXiv:2510.13747, 2025

    Wenwen Tong, Hewei Guo, Dongchuan Ran, Jiangnan Chen, Jiefan Lu, Kaibin Wang, Keqiang Li, Xiaoxu Zhu, Jiakui Li, Kehan Li, et al. Interactiveomni: A unified omni-modal model for audio-visual multi-turn dialogue.arXiv preprint arXiv:2510.13747, 2025

    arXiv 2025

  33. [33]

    Socialomni: Benchmarking audio-visual social interactivity in omni models.arXiv preprint arXiv:2603.16859, 2026

    Tianyu Xie, Jinfa Huang, Yuexiao Ma, Rongfang Luo, Yan Yang, Wang Chen, Yuhui Zeng, Ruize Fang, Yixuan Zou, Xiawu Zheng, et al. Socialomni: Benchmarking audio-visual social interactivity in omni models.arXiv preprint arXiv:2603.16859, 2026

    Pith/arXiv arXiv 2026

  34. [34]

    Mini-omni: Language models can hear, talk while thinking in streaming.arXiv preprint arXiv:2408.16725, 2024

    Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming.arXiv preprint arXiv:2408.16725, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities.arXiv preprint arXiv:2410.11190, 2024

    Zhifei Xie and Changqiao Wu. Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities.arXiv preprint arXiv:2410.11190, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Qwen3-omni technical report.arXiv preprint arXiv:2509.17765, 2025

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-omni technical report.arXiv preprint arXiv:2509.17765, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model

    Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Ponnuswamy Sadayappan, Xia Hu, et al. Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 19803–19813, 2025

    2025

  38. [38]

    Omni-autothink: Adaptive multimodal reasoning via reinforcement learning.arXiv preprint arXiv:2512.03783, 2025

    Dongchao Yang, Songxiang Liu, Disong Wang, Yuanyuan Wang, Guanglu Wan, and Helen Meng. Omni-autothink: Adaptive multimodal reasoning via reinforcement learning.arXiv preprint arXiv:2512.03783, 2025

    arXiv 2025

  39. [39]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19792–19802, 2025

    2025

  40. [40]

    Fit and prune: Fast and training-free visual token pruning for multi-modal large language models

    Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training-free visual token pruning for multi-modal large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 22128–22136, 2025

    2025

  41. [41]

    Full Tokens

    Ziwei Zhou, Rui Wang, Zuxuan Wu, and Yu-Gang Jiang. Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities.arXiv preprint arXiv:2505.17862, 2025. 12 A Implementation Details This section provides additional implementation details omitted from the main text, with an emphasis on the budget calibration protocol and default hyper...

    arXiv 2025

  42. [42]

    Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or ...