Pith. sign in

REVIEW 4 major objections 4 minor 69 references

SAMOT: State-Aware Step Modulation and Optimal Transport Matching for Audio-Visual Instance Segmentation

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A framework with adaptive Mamba step sizes and optimal-transport matching achieves state-of-the-art results on the AVISeg benchmark, beating the prior best by +3.76 FSLA, +2.75 HOTA, and +2.58 mAP under MS-COCO pretraining.

desk verdict Solid, honest AVIS paper with a real mechanism gap: the ADSM-to-Mamba equation is asserted but never written; everything else points to a legitimate SOTA result that deserves a normal peer review. read the letter →

arxiv 2608.08147 v1 pith:EOL5K3N5 submitted 2026-08-08 cs.MM

Kai Peng , Yunzhe Shen , Miao Zhang , Leiye Liu , Wei Ji , Jingjing Li , Yongri Piao , Huchuan Lu This is my paper · ORCID
classification cs.MM
keywords Audio-visualinstancesegmentationStatespacemodelMambaOptimaltransportSinkhorniterationsMultimodalmatchingVideoTemporalmodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper addresses audio-visual instance segmentation, the task of classifying, segmenting, and tracking every sounding object in a video. It argues that existing fixed-step Transformers and Mamba models cannot handle modality-state changes in long sequences, and that implicit cross-attention cannot resolve structural and distributional differences between audio and video. To fix this, the paper proposes SAMOT with Adaptive Dynamic Step Modulation, which adjusts the Mamba state-update step size based on temporal and cross-modal signals, and Optimal Transport-based Matching Modulation, which aligns audio and video queries by solving an entropy-regularized optimal transport problem with spatial and temporal priors plus an MMD regularizer. On the AVISeg benchmark the framework reports state-of-the-art results, beating the previous best by +3.76 FSLA, +2.75 HOTA, and +2.58 mAP under MS-COCO pretraining.

What carries the argument

The machinery is a pair of modules inserted into a Mamba-based audio-visual segmentation pipeline. ADSM computes two scalar indicators of the current modality state: $s_{\text{modality}}$ (max-norm temporal change of audio-conditioned and raw visual features) and $s_{\text{diff}}$ (cosine distance between them), then uses a $\Delta$ Gating Router to combine three tanh branches—current audio-conditioned visual, current audio, and historical context—into a step-size offset $d$ that is added to the fixed base step $\Delta_0$ and injected into the selective state-space discretization. OT-MM builds a transport cost $C_t$ from normalized feature distance plus a relative positional prior $\beta |i-j| / \max(N_a, N_v)$, warms the log-kernel with the previous transport plan as a temporal consistency prior, solves the entropy-regularized problem in the log domain with Sinkhorn iterations, projects audio queries through the transport plan, and regularizes the resulting distributions with a multi-kernel MMD loss.

What would settle it

If a variant that keeps the base Mamba step size fixed and instead multiplies the audio-conditioned visual features by a per-frame learned gate reproduces the same FSLA, HOTA, and mAP gains on AVISeg, then the step-size injection is not the operative mechanism and the ADSM claim collapses.

Watch

Extended reading notes

Core claim

The central claim is that the two bottlenecks in audio-visual instance segmentation—complex modality-state changes and cross-modal distribution mismatch—can be addressed by making the temporal state-space model input-adaptive and by making cross-modal association an explicit matching problem. ADSM estimates a temporal variation score and a cross-modal discrepancy score, feeds them through a $\Delta$ Gating Router that weights visual, audio, and historical branches, and adds the resulting offset to the base Mamba step size, so the model updates faster at modality transitions and stays stable otherwise. OT-MM defines a cost matrix combining cosine feature distance with a relative positional prior, reuses the previous time step's transport plan as a temporal consistency prior, solves the entropy-regularized problem with log-domain Sinkhorn iterations, and adds an MMD loss to align the matched audio and video distributions. The paper reports that this combination achieves state-of-the-art performance on AVISeg, with FSLA 48.18, HOTA 67.27, and mAP 47.62 under MS-COCO pretraining, surpassing AVISM by +3.76, +2.75, and +2.58 respectively.

Load-bearing premise

The load-bearing premise is that adding the scalar offset $\Delta$ to the base step size inside the Mamba discretization is what actually changes the state update in the trained model, and that the reported gains come from that mechanism rather than from the extra feature reweighting; the paper never writes the resulting recurrence, so this premise is unverified.

Editorial extensions

If this is right

  • If ADSM works as claimed, state-space models for long-video audio-visual understanding can remain stable during steady segments while reacting quickly to sound-source switches or audio dropouts.
  • If OT-MM works as claimed, explicit optimal-transport matching with positional and temporal priors yields more reliable instance-level audio-visual correspondence than implicit cross-attention or contrastive learning.
  • The reported gains on FSLAs and FSLAm suggest the framework is especially stronger in silent and multi-source scenarios, which are the hard cases for AVIS.
  • The online (causal) variant retaining competitive performance suggests the same principles could transfer to streaming inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same ADSM recipe—per-step state indicators routed into a step-size offset—could be applied to other state-space-model multimodal tasks such as audio-visual event localization or video question answering, where modality-state changes also break long-range modeling.
  • The log-domain Sinkhorn matching with a temporal prior and MMD regularizer could serve as a generic alignment layer for any tracking-by-matching problem with heterogeneous query sets, not just sound-video pairs.
  • A testable extension: replacing the manually fixed priors $\beta$, $\gamma$, and $\epsilon$ with learned or scheduled values could further improve the transport plan when instance counts vary widely across frames.
  • The paper's Fig. 9 suggests a causal variant; an explicit benchmark of streaming latency versus offline accuracy would clarify deployment potential.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SAMOT, a framework for audio-visual instance segmentation (AVIS) that combines Adaptive Dynamic Step Modulation (ADSM) for Mamba-based temporal modeling and Optimal Transport-based Matching Modulation (OT-MM) with MMD regularization for cross-modal instance-level association. The method is evaluated on the AVISeg benchmark and claims state-of-the-art results, with reported gains of +3.76 FSLA, +2.75 HOTA, and +2.58 mAP over AVISM under MS-COCO pretraining, supported by module ablations, qualitative visualizations, and an efficiency analysis. The code and model are made publicly available.

Significance. If the reported results are reproducible, SAMOT would be the current best performer on AVISeg, and the two proposed mechanisms address real limitations of fixed-step Transformers and implicit matching in long-sequence audio-visual modeling. The paper's strengths include a public code release, a systematic set of ablations for the main components (Tables 2-5), and an efficiency comparison (Fig. 7). However, the central mechanistic claim about ADSM is asserted but not formally derived, the ablation baseline is numerically different from the official baseline used in the headline comparison, and all results come from single runs without error bars. These issues currently leave the attribution of the reported gains to the proposed mechanisms insufficiently supported.

major comments (4)
  1. [Sec. 3.2, Eq. (6)] The statement that the adjusted delta is 'directly injected into the discretization step of the selective state-space mechanism' is not substantiated by any equation in the main text. The paper never specifies how delta enters the discrete-time recursion (e.g., A_bar_t = exp(delta_t A) and B_bar_t = (delta_t A)^{-1}(exp(delta_t A) - I) delta_t B), nor does it define the relationship between Delta_0 and the Mamba per-channel time-scale parameter, nor the behavior when Delta_0 + d is non-positive. Without this specification, the implementation could act as a gating or scaling heuristic rather than true step-size modulation, and the attribution of the +3.76 FSLA gain to genuinely adaptive discretization is unsupported. This is a load-bearing gap between Eq. (6) and the claim that follows it.
  2. [Sec. 4.1, Tables 1 and 2] The headline comparison in Table 1 uses the official AVISM baseline, while the module ablation in Table 2 uses a reproduced baseline that includes the hierarchical temporal tracker but omits ADSM and OT-MM. The paper explicitly notes that these baselines are not numerically identical, but it does not report where the reproduced baseline sits relative to the official AVISM number. Consequently, the incremental improvements in Table 2 (e.g., +2.62 FSLA from adding ADSM and OT-MM to the reproduced baseline) cannot be reconciled with the +3.76 FSLA gain over official AVISM in Table 1. The authors should either include the reproduced baseline in Table 1 or provide the numerical relationship between the two baselines; otherwise the attribution of the headline gains to the individual modules is not directly verifiable.
  3. [Sec. 4.1, Table 1] All experimental results are reported from single runs with no standard deviations, confidence intervals, or significance tests. This is particularly concerning for the standard-setting comparison, where the HOTA gain over AVISM is only +0.85; such a difference is within plausible run-to-run variation for instance segmentation methods trained with batch size 1. The authors should report results over multiple seeds (or at least state the number of runs) to support the state-of-the-art claim.
  4. [Sec. 3.3, Eqs. (8)-(9)] The log-domain Sinkhorn updates are essential for OT-MM, but the 'detailed derivations' are deferred to a supplementary file that is not included in the preprint. The reader therefore cannot verify that the temporal prior in Eq. (8) preserves the desired marginal constraints, how M^{t-1} is initialized at the start of a sequence, or why L=10 Sinkhorn iterations are sufficient. Please include these derivations in the manuscript or provide a public version of the supplementary that is referenced with a stable link.
minor comments (4)
  1. [Abstract and Intro] The phrase 'methods performing implicit matching ignore the inherent distributional inconsistencies' is overly broad; implicit matching methods such as cross-attention may reduce semantic gaps without explicitly modeling distributional discrepancies, and the sentence could be rephrased to make that distinction.
  2. [Sec. 2.2] The sentence 'AV-Mamba [25] integrate temporal and cross-modal selective mechanisms' has a subject-verb agreement error; it should be 'integrates'.
  3. [Fig. 6] The numeric labels inside the bar chart are very small and hard to read; a table or larger font would make the comparison of alternative distance formulations clearer.
  4. [References] Reference [3] cites the arXiv version of Mask2Former for Video Instance Segmentation; if a peer-reviewed version exists, it should be cited instead.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the reported gains are empirical benchmark results, and the ADSM/OT-MM components are proposed mechanisms rather than fitted parameters disguised as predictions.

full rationale

We examined the derivation chain from Eqs. (1)-(11) and the experimental protocol. The central result is the benchmark comparison in Table 1, where SAMOT is trained on AVISeg and evaluated under the same protocol as prior methods; no parameter is fitted to a subset of the evaluation data and then reported as a prediction of the same quantity. ADSM's Delta = Delta_0 + d (Eq. 6) is a module output whose use in the Mamba discretization is asserted but never written as a state-space recursion; that is an underspecification or correctness risk, not a circular step, because the claim does not reduce to its inputs by construction. OT-MM formulates matching as entropy-regularized optimal transport and adds an MMD regularizer; these are standard components with no hidden fitted input. The ablation baseline is explicitly stated to be a reproduced framework rather than the official AVISM baseline, so the reported gains are not a tautology. Self-citations appear in related work (e.g., Refs. [12], [44], [46]) but are not load-bearing: no uniqueness theorem is invoked, and the method's components are defined by the paper's own equations. We therefore find no significant circularity.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The framework relies on standard pretrained encoders (ResNet-50, VGGish), the Mamba state-space formulation, and the AVISeg benchmark annotations. Hyperparameters alpha, beta, epsilon, gamma, eta, L, window size w, and MMD bandwidths are hand-set. No new physical or conceptual entities are introduced.

free parameters (8)
  • alpha = 0.2
    Controls intensity of cross-modal fusion in Eq. (1); set by hand.
  • beta = 0.1
    Weight for relative positional prior in OT cost, Eq. (7); set by hand.
  • epsilon = 0.01
    Entropic regularization coefficient in Sinkhorn, Eq. (8); set by hand.
  • gamma = 0.1
    Momentum for temporal consistency prior, Eq. (8); set by hand.
  • eta = 1e-8
    Small constant to prevent invalid logarithm in Eq. (8); set by hand.
  • L = 10
    Number of Sinkhorn iterations; set by hand.
  • window_size_w = 3
    Local temporal window in hierarchical tracker; chosen by sweep in Fig. 8.
  • MMD_bandwidths = S={0.1, 1.0, 10.0}
    Multi-scale Gaussian bandwidth set for MMD loss, Eq. (11); fixed by design.
assumptions (4)
  • standard math Mamba selective state space model equations from Gu and Dao are correct and applicable.
    The paper builds all temporal modeling on Mamba without re-deriving its state transition equations.
  • domain assumption Uniform marginal constraints in the optimal transport formulation are appropriate for audio-visual instance matching.
    The transport problem enforces balanced assignment between audio and video queries, which may not reflect unequal instance counts.
  • domain assumption VGGish pretrained on AudioSet provides audio features suitable for AVIS.
    The audio encoder is adopted without fine-tuning or adaptation analysis.
  • domain assumption The AVISeg benchmark annotations and metrics are reliable for evaluating AVIS.
    The paper relies entirely on this benchmark for all conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAMOT: State-Aware Step Modulation and Optimal Transport Matching for Audio-Visual Instance Segmentation." pith.science (2026). https://pith.science/paper/EOL5K3N5

@misc{pith2026260808147,
  author       = {Pith},
  title        = {Pith review of: SAMOT: State-Aware Step Modulation and Optimal Transport Matching for Audio-Visual Instance Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EOL5K3N5}},
  note         = {Machine review of arXiv:2608.08147}
}
read the original abstract

Audio-Visual Instance Segmentation (AVIS) aims to simultaneously classify, segment, and track sounding objects within video sequences. Unlike Audio-Visual Semantic Segmentation (AVS), AVIS involves instance-level modeling across longer video sequences, introducing two key challenges: (1) complex modality-state changes disrupt long-range modeling, and (2) substantial structural and distributional discrepancies between modalities hinder precise instance-level association. Existing methods rely on fixed-step Transformers and recursive Mamba models, lacking adaptability to modality-state changes. In addition, methods performing implicit matching ignore the inherent distributional inconsistencies. To address these issues, we propose a framework with Adaptive Dynamic Step Modulation (ADSM) and Optimal Transport-based Matching Modulation (OT-MM). ADSM adaptively modulates Mamba step sizes using temporal variation, cross-modal discrepancy, and historical context, balancing rapid response to modality-state changes with stable long-range modeling. OT-MM explicitly formulates instance-level cross-modal matching as an entropy-regularized optimal transport problem solved via log-domain Sinkhorn iterations, and further enforces distribution-level coherence with an MMD regularizer. Extensive experiments demonstrate state-of-the-art performance on the AVIS benchmark (+3.76 FSLA, +2.75 HOTA, +2.58 mAP), verified through comprehensive qualitative visualizations. The code and model are available at https://github.com/happylife-pk/SAMOT.

Figures

Figures reproduced from arXiv: 2608.08147 by the authors.

Figure 1
Figure 1. Illustration of the two core challenges in AVIS. (a) Al [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Overall architecture of the proposed framework. (b) ADSM dynamically modulates Mamba step sizes using [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison between AVISM and Ours across diverse audio scenarios with varying sound sources. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Visualization of the effective step size [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Visualization of cross-modal feature distributions [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of alternative distance formulations [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: Comparison between offline and online settings. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 8
Figure 8. Figure 8: Effect of window size on performance and efficiency [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 39 canonical work pages

  1. [1]

    Rong Chao, Wenze Ren, You-Jin Li, Kuo-Hsuan Hung, Sung-Feng Huang, Szu-Wei Fu, Wen-Huang Cheng, and Yu Tsao. 2025. Leveraging Mamba with Full-Face Vision for Audio-Visual Speech Enhancement.arXiv preprint arXiv:2508.13624 (2025)

    arXiv 2025

  2. [2]

    Yuanhong Chen, Chong Wang, Yuyuan Liu, Hu Wang, and Gustavo Carneiro

    work page

  3. [3]

    Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G. Schwing. 2021. Mask2Former for Video Instance Segmentation. arXiv:2112.10764 [cs.CV] https://arxiv.org/abs/2112.10764

    arXiv 2021

  4. [4]

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. 2022. Masked-attention mask transformer for universal image segmen- tation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1290–1299

    2022

  5. [5]

    Ying Cheng, Yang Li, Junjie He, and Rui Feng. 2024. Mixtures of experts for audio-visual learning.Advances in Neural Information Processing Systems37 (2024), 219–243

    work page 2024

  6. [6]

    Yashar Deldjoo, Mihai Gabriel Constantin, Hamid Eghbal-Zadeh, Bogdan Ionescu, Markus Schedl, and Paolo Cremonesi. 2018. Audio-visual encoding of multimedia content for enhancing movie recommendations. InProceedings of the 12th ACM conference on recommender systems. 455–459

    work page 2018

  7. [7]

    Mehmet Hamza Erol, Arda Senocak, Jiu Feng, and Joon Son Chung. 2024. Audio mamba: Bidirectional state space model for audio representation learning.IEEE Signal Processing Letters(2024)

    work page 2024

  8. [8]

    Hao Fang, Tong Zhang, Xiaofei Zhou, and Xinxin Zhang. 2024. Learning better video query with sam for video instance segmentation.IEEE Transactions on Circuits and Systems for Video Technology(2024)

    work page 2024

Show all 69 references
  1. [9]

    Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. 2024. Avseg- former: Audio-visual segmentation with transformer. InProceedings of the AAAI conference on artificial intelligence, Vol. 38. 12155–12163

    2024

  2. [10]

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In2017 IEEE international conference on acoustics, speech and signal process...

    2017

  3. [11]

    Sitong Gong, Tianyu Yan, Caixin Kang, Bo Zheng, Xiang Ruan, Huchuan Lu, Kaipeng Zhang, Yoichi Sato, and Yifei Huang. 2026. Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos.arXiv preprint arXiv:2607.11523 (2026)

    2026 arXiv

  4. [12]

    Sitong Gong, Yunzhi Zhuge, Lu Zhang, Yifan Wang, Pingping Zhang, Lijun Wang, and Huchuan Lu. 2025. AVS-Mamba: Exploring Temporal and Multi- Modal Mamba for Audio-Visual Segmentation.IEEE Transactions on Multimedia 27 (2025), 5413–5425. doi:10.1109/TMM.2025.3542995

    2025

  5. [13]

    Sitong Gong, Yunzhi Zhuge, Lu Zhang, Zongxin Yang, Pingping Zhang, and Huchuan Lu. 2025. The Devil Is in Temporal Token: High Quality Video Reasoning Segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 29183–29192. doi:10.1109/CVP...

    2025

  6. [14]

    Sitong Gong, Yunzhi Zhuge, Lu Zhang, Jiazuo Yu, Pingping Zhang, Xu Jia, and Huchuan Lu. 2026. Reinforcing Video Object Segmentation to Think Before It Segments. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3835–3844

    2026

  7. [15]

    Sitong Gong, Yunzhi Zhuge, Lu Zhang, Pingping Zhang, and Huchuan Lu. 2025. Complementary and Contrastive Learning for Audio-Visual Segmentation.IEEE Transactions on Multimedia27 (2025), 7407–7418. doi:10.1109/TMM.2025.3599048

    2025

  8. [16]

    Albert Gu and Tri Dao. 2023. Mamba: Linear-Time Sequence Modeling with Selective State Spaces.arXiv preprint arXiv:2312.00752(2023)

    2023 arXiv

  9. [17]

    Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. 2022. On the parame- terization and initialization of diagonal state space models.Advances in Neural Information Processing Systems35 (2022), 35971–35983

    2022

  10. [18]

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. 2021. Combining recurrent, convolutional, and continuous-time models with linear state space layers.Advances in neural information processing systems34 (2021), 572–585

    2021

  11. [19]

    Ruohao Guo, Xianghua Ying, Yaru Chen, Dantong Niu, Guangyao Li, Liao Qu, Yanyu Qi, Jinxing Zhou, Bowei Xing, Wenzhen Yue, et al. 2025. Audio-visual instance segmentation. InProceedings of the Computer Vision and Pattern Recog- nition Conference. 13550–13560

    2025

  12. [20]

    Dawei Hao, Yuxin Mao, Bowen He, Xiaodong Han, Yuchao Dai, and Yiran Zhong

  13. [21]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition. 770–778

    2016

  14. [22]

    In Proceedings of the AAAI conference on artificial intelligence, Vol

    Improving audio-visual segmentation with bidirectional generation. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38. 2067–2075

  15. [23]

    Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, s...

    2017

  16. [24]

    Miran Heo, Sukjun Hwang, Seoung Wug Oh, Joon-Young Lee, and Seon Joo Kim

  17. [25]

    Ziru Huang, Jia Li, Wenjie Zhao, Yunhui Guo, and Yapeng Tian. 2024. AV- Mamba: Cross-Modality Selective State Space Models for Audio-Visual Question Answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW). 1–4

    2024

  18. [26]

    Haichao Jiang, Tianming Liang, Wei-Shi Zheng, and Jian-Fang Hu. 2026. Refer- Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 39...

    2026

  19. [27]

    Shaofei Huang, Rui Ling, Tianrui Hui, Hongyu Li, Xu Zhou, Shifeng Zhang, Si Liu, Richang Hong, and Meng Wang. 2025. Revisiting audio-visual segmentation with vision-centric transformer. InProceedings of the Computer Vision and Pattern Recognition Conference. 8352–8361

    2025

  20. [28]

    Kexin Li, Zongxin Yang, Lei Chen, Yi Yang, and Jun Xiao. 2023. Catr: Combinatorial-dependence audio-queried transformer for audio-visual video segmentation. InProceedings of the 31st ACM international conference on multi- media. 1485–1494

    2023

  21. [29]

    Songtao Li and Hao Tang. 2024. Multimodal alignment and fusion: A survey. arXiv preprint arXiv:2411.17040(2024)

    2024

  22. [30]

    Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. 2024. Videomamba: State space model for efficient video understanding. In European conference on computer vision. Springer, 237–255

    2024

  23. [31]

    Tianming Liang, Qirui Du, Jian-Fang Hu, Haichao Jiang, Zicheng Lin, and Wei- Shi Zheng. 2026. Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search. InForty-third International Conference on Machine Learning. https://openreview.net/forum?id=H0tGv2kBi8

    2026

  24. [32]

    Tianming Liang, Haichao Jiang, Yuting Yang, Chaolei Tan, Shuai Li, Wei-Shi Zheng, and Jian-Fang Hu. 2026. Long-RVOS: A Comprehensive Benchmark for Long-Term Referring Video Object Segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2026

  25. [33]

    Yan Li, Yifei Xing, Xiangyuan Lan, Xin Li, Haifeng Chen, and Dongmei Jiang

  26. [34]

    Xinying Lin, Xuyang Liu, Yiyu Wang, Teng Ma, and Wenqi Ren. 2026. V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Lan- guage Models.arXiv preprint arXiv:2603.27650(2026)

    2026

  27. [35]

    Leiye Liu, Miao Zhang, Jihao Yin, Tingwei Liu, Wei Ji, Yongri Piao, and Huchuan Lu. 2025. Defmamba: Deformable visual state space model. InProceedings of the Computer Vision and Pattern Recognition Conference. 8838–8847

    2025

  28. [36]

    Xuyang Liu, Xiyan Gui, Yuchao Zhang, and Linfeng Zhang. 2026. Mixing Im- portance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models. InInternational Conference on Learning Representations. https://openreview.net/forum?id=B2iqbCQviR

    2026

  29. [37]

    Jingyang Lin, Xinru Ying, Jiaqi Mo, Lina Wei, Fangfang Wang, Canghong Jin, and Guanlin Chen. 2025. Mamba-Based Multimodal Continual Learning for Audio- Visual Classification with Prototype-Enhanced Anti-Forgetting Mechanism. In Proceedings of the 3rd International Workshop on ...

    2025

  30. [38]

    Xuyang Liu, Ziming Wang, Junjie Chen, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Siteng Huang, and Honggang Chen. 2026. Global Compression Com- mander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision- Language Models. InProceedings of the AAAI Confer...

    2026 doi

  31. [39]

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. 2024. Vmamba: Visual state space model. Advances in neural information processing systems37 (2024), 103031–103063

    2024

  32. [40]

    Lidong Lu, Guo Chen, Zhu Wei, Zhiqi Li, Yicheng Liu, and Tong Lu. 2026. AV- Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 33477–33487. MM ’26, November 10...

    2026

  33. [41]

    Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang. 2025. Video Com- pression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computatio...

    2025

  34. [42]

    Juncheng Ma, Peiwen Sun, Yaoting Wang, and Di Hu. 2024. Stepping stones: a progressive training strategy for audio-visual semantic segmentation. InEuropean Conference on Computer Vision. Springer, 311–327

    2024

  35. [43]

    Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research9, Nov (2008), 2579–2605

    2008

  36. [44]

    Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Yidong Han, Wei Ji, Jingjing Li, Yongri Piao, and Huchuan Lu. 2026. Selective noise suppression and discrimina- tive mutual interaction for robust audio-visual segmentation.IEEE Transactions on Multimedia(2026)

    2026

  37. [45]

    Jinbae Seo, Hyeongjun Kwon, Kwonyoung Kim, Jiyoung Lee, and Kwanghoon Sohn. 2026. Learning What to Hear: Boosting Sound-Source Association for Ro- bust Audiovisual Instance Segmentation. InICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processin...

    2026

  38. [46]

    Lidong Lu, Guo Chen, Zhu Wei, Yicheng Liu, and Tong Lu. 2025. Learning Visual Affordance from Audio.arXiv preprint arXiv:2512.02005(2025)

    2025

  39. [47]

    Jian Shi, Baoli Sun, Xinchen Ye, Zhihui Wang, Xiaolong Luo, Jin Liu, Heli Gao, and Haojie Li. 2023. Semantic Decomposition Network With Contrastive and Structural Constraints for Dental Plaque Segmentation.IEEE Trans. Medical Imaging42, 4 (2023), 935–946. doi:10.1109/TMI.2022.3221529

    2023

  40. [48]

    Peiwen Sun, Honggang Zhang, and Di Hu. 2024. Unveiling and Mitigating Bias in Audio Visual Segmentation. InProceedings of the 32nd ACM International Conference on Multimedia. 7259–7268

    2024

  41. [49]

    Jia Wang, Yunan Mei, Jiarui Liu, and Xin Fan. 2025. SAM-Guided Robust Repre- sentation Learning for One-Shot 3D Medical Image Segmentation.arXiv preprint arXiv:2504.20501(2025)

    2025 arXiv

  42. [50]

    Jia Wang, Yuchen Sun, Yunan Mei, Zihao Xu, and Xin Fan. 2025. Personalizing Federated Learning Guided by Site-Aggregated Representation for Multi-Site One-Shot Medical Image Segmentation. In2025 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 7113–7120

    2025

  43. [51]

    Yunzhe Shen, Kai Peng, Leiye Liu, Wei Ji, Jingjing Li, Miao Zhang, Yongri Piao, and Huchuan Lu. 2025. Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation.arXiv preprint arXiv:2509.18912(2025)

    2025

  44. [52]

    Junfeng Wu, Yi Jiang, Song Bai, Wenqing Zhang, and Xiang Bai. 2022. Seqformer: Sequential transformer for video instance segmentation. InEuropean Conference on Computer Vision. Springer, 553–569

    2022

  45. [53]

    Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, and Changsheng Xu. 2025. Towards Visual Grounding: A Survey.IEEE Transactions on Pattern Analysis and Machine Intelligence(2025). doi:10.1109/TPAMI.2025.3630635

    2025

  46. [54]

    Zhenghao Xing, Hao Chen, Binzhu Xie, Jiaqi Xu, Ziyu Guo, Xuemiao Xu, Jianye Hao, Chi-Wing Fu, Xiaowei Hu, and Pheng-Ann Heng. 2025. EchoTraffic: En- hancing traffic anomaly understanding with audio-visual insights. InProceedings of the Computer Vision and Pattern Recognition C...

    2025

  47. [55]

    Qi Yang, Xing Nie, Tong Li, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan, and Shiming Xiang. 2024. Cooperation does matter: Exploring multi-order bilateral relations for audio-visual segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  48. [56]

    Xin-Cheng Wen, Jiaxin Ye, Yan Luo, Yong Xu, Xuan-Ze Wang, Chang-Li Wu, and Kun-Hong Liu. 2022. CTL-MTNet: A Novel CapsNet and Transfer Learning- Based Mixed Task Net for the Single-Corpus and Cross-Corpus Speech Emotion Recognition. InProceedings of the Thirty-First Internatio...

    2022 doi

  49. [57]

    Jiaxin Ye, Xin-Cheng Wen, Xuan-Ze Wang, Yong Xu, Yan Luo, Chang-Li Wu, Li-Yan Chen, and Kunhong Liu. 2022. GM-TCNet: Gated Multi-Scale Temporal Convolutional Network Using Emotion Causality for Speech Emotion Recogni- tion.Speech Communication145 (2022), 21–35. doi:10.1016/j.s...

    2022 doi

  50. [58]

    Tao Zhang, Xingye Tian, Yu Wu, Shunping Ji, Xuebo Wang, Yuan Zhang, and Pengfei Wan. 2023. Dvis: Decoupled video instance segmentation framework. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1282– 1291

    2023

  51. [59]

    Yan Zheng, Jeffrey Jestes, Jeff M Phillips, and Feifei Li. 2013. Quality and efficiency for kernel density estimates in large data. InProceedings of the 2013 ACM SIGMOD International Conference on Management of Data. 433–444

    2013

  52. [60]

    Jialong Zhong, Tingwei Liu, Baokun Yue, Jingjing Li, Yongri Piao, Miao Zhang, Leiye Liu, Jiahong Jiang, Wei Ji, and Huchuan Lu. 2026. AdaSurvMamba: Dynamic Fusion and Semantic Scanning for Multimodal Survival Analysis.arXiv preprint arXiv:2607.16260(2026)

    2026 arXiv

  53. [61]

    Shusheng Yang, Xinggang Wang, Yu Li, Yuxin Fang, Jiemin Fang, Wenyu Liu, Xun Zhao, and Ying Shan. 2022. Temporally efficient vision transformer for video instance segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2885–2895

    2022

  54. [62]

    Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. 2022. Audio– visual segmentation. InEuropean Conference on Computer Vision. Springer, 386– 403

    2022

  55. [63]

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2021. Deformable DETR: Deformable Transformers for End-to-End Object Detection. InInternational Conference on Learning Representations

    2021

  56. [64]

    Yunzhi Zhuge, Sitong Gong, Lu Zhang, Qi Xu, Wenda Zhao, Jin Zhan, and Huchuan Lu. 2026. Context-Infused Trajectories: Enhancing Context and Frame Consistency in Reasoning Video Object Segmentation.IEEE Transactions on Image Processing35 (2026), 5239–5252. doi:10.1109/TIP.2026.3689427

    2026

  57. [66]

    Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, et al. 2025. Audio- visual segmentation with semantics.International Journal of Computer Vision 133, 4 (2025), 1644–1664

    2025

  58. [1924]

    doi:10.18653/v1/2025.emnlp-main.98

    2025 doi

  59. [2022]

    Vita: Video instance segmentation via object token association.Advances in neural information processing systems35 (2022), 23109–23120

    2022

  60. [2024]

    InEuropean Conference on Computer Vision

    Cpm: Class-conditional prompting machine for audio-visual segmentation. InEuropean Conference on Computer Vision. Springer, 438–456

  61. [2025]

    InProceedings of the Computer Vision and Pattern Recognition Conference

    Alignmamba: Enhancing multimodal mamba with local and global cross- modal alignment. InProceedings of the Computer Vision and Pattern Recognition Conference. 24774–24784

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.