Pith. sign in

REVIEW 3 major objections 86 references

STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation

T0 review · 3 major / 0 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-space recurrence first makes video token redundancy measurable, then lets a model throw away 85% of tokens and still beat uncompressed baselines on reasoning segmentation.

desk verdict Clean engineering paper: enrich with decoupled causal SSM first, then task-grounded hierarchical compression; 85% reduction with competitive zero-shot numbers, but single-run tables and an online–offline gap keep the headline claim provisional. read the letter →

arxiv 2607.02922 v1 pith:MC7XFU7C submitted 2026-07-03 cs.CV

Syed Ariff Syed Hesham , Yun Liu , Guolei Sun , Jing Yang , Henghui Ding , Xue Geng , Xudong Jiang This is my paper
classification cs.CV
keywords videoreasoningsegmentationtokencompressionstate-spacemodelsMambaspatiotemporalmodelingstreamingmultimodalLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long video reasoning segmentation produces so many visual tokens that quadratic attention becomes impractical. Most compression schemes decide what to keep from raw encoder features that have never seen other frames, so they cannot tell semantic redundancy from surface similarity. The paper shows that a linear state-space recurrence can first condition every token on temporal context at linear cost; the resulting near-identical enriched states themselves become a reliable redundancy signal. STAC uses that signal: it enriches features with bidirectional spatial and causal temporal scans, then applies hierarchical adaptive compression whose thresholds are trained end-to-end for mask accuracy. The result is roughly 85 percent fewer tokens, 1.8 times faster inference, streaming compatibility, and higher scores than full-token baselines on both referring and zero-shot reasoning benchmarks.

What carries the argument

State-informed Spatiotemporal Aggregator followed by Hierarchical State-adaptive Compression: bidirectional spatial Mamba scans plus causal temporal Mamba scans enrich tokens so that cosine similarity of the resulting states can drive adaptive temporal-then-spatial merging whose thresholds are optimised by segmentation loss via straight-through estimation.

What would settle it

On ReasonVOS-style queries that require future context, measure whether online STAC (causal only) drops more than a few J&F points relative to an offline bidirectional version while still matching or beating full-token baselines; a large online gap that also undercuts the full-token score would falsify the claim that the causal redundancy signal is sufficient.

Watch

Extended reading notes

Core claim

When encoder features are first passed through selective state-space recurrence, the recurrence itself produces a feature space in which content redundancy becomes directly measurable; hierarchical compression performed in that space can discard about 85 percent of visual tokens while still outperforming models that keep every token, and the causal half of the design supports online streaming.

Load-bearing premise

That the near-identical states produced by causal recurrence are a trustworthy enough redundancy signal that adaptive thresholds can safely discard most tokens without losing the motion or future-frame cues needed for complex reasoning.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. STAC addresses the quadratic cost of dense video tokens in reasoning segmentation by placing selective state-space enrichment before hierarchical compression. Encoder features are first processed by a State-informed Spatiotemporal Aggregator (SSA) that applies bidirectional spatial Mamba scans within frames and causal temporal scans across frames (Eqs. 1–3), producing representations in which content redundancy is measurable. Hierarchical State-adaptive Compression (HSC) then performs temporal-then-spatial reduction with EMA-based adaptive thresholds (Eqs. 4–6), and the discrete retention decisions are trained end-to-end with a segmentation objective via straight-through estimation (Eqs. 7–8). Trained only on referring data (MeViS, Ref-YouTube-VOS), the method reports ~85% token reduction, ~1.8× speedup, and competitive or superior J&F on Ref-DAVIS17, MeViS, Ref-YouTube-VOS, ReasonVOS and ReVOS in a zero-shot, streaming-compatible setting (Table 2).

Significance. If the results hold under multi-seed verification, the paper supplies a practical and architecturally clean solution to a genuine bottleneck: long-video reasoning segmentation under quadratic attention. The explicit decoupling of bidirectional spatial from causal temporal scanning is a useful design principle that simultaneously enables streaming and respects the distinct causal structure of the two axes. Task-grounded optimisation of compression thresholds via STE, the clean zero-shot isolation of reasoning benchmarks, and the public code release are concrete strengths. Ablations (Tables 3–4, Fig. 3) consistently favour the claimed design choices (temporal-first ordering, merge over prune, bidirectional spatial + causal temporal). The work is therefore of clear interest to the video-MLLM and efficient-vision communities.

major comments (3)
  1. The central claim that STAC “surpasses compression-free baselines … under 85% token reduction” (abstract, §4.2, Table 2) rests on single-run point estimates with no multi-seed statistics, error bars or confidence intervals. On ReasonVOS the reported margin over GLUS is only +2.5 J&F; the paper itself records a 3.5-point online–offline gap (48.8 vs 52.3, §4.3). Without variance estimates it is impossible to judge whether the “surpassing” statement is statistically secure.
  2. Table 2 and §4.2 compare STAC (15% tokens) against full-token baselines (GLUS, VISA, VideoLISA, etc.) without a matched-token-budget re-run of the strongest full-token method. Consequently it remains unclear whether the gains arise from the SSM-derived redundancy signal (Eqs. 2–3, 4–6) or simply from any aggressive compression that forces the model to discard static background. A controlled ablation that forces GLUS (or an equivalent) to the same token budget would isolate the contribution of the proposed enrichment-before-compression pipeline.
  3. The weakest modelling assumption—that cosine similarities of causal-SSM states constitute a reliable, task-aligned redundancy signal—is only partially stress-tested. The online–offline gap on ReasonVOS (§4.3) already shows that future-frame information matters for reasoning queries; yet the paper provides no quantitative analysis of which motion or multi-hop tokens are discarded by the adaptive thresholds, nor any failure-case study on long-horizon causal reasoning. Strengthening this analysis would make the claim that the recurrence-derived signal is “task-aligned” more convincing.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical architecture paper whose compression policy and J&F gains are measured on held-out public benchmarks, not forced by definition or self-citation.

full rationale

STAC is a standard empirical CV method paper. The core chain (SSA enrichment via decoupled Mamba scans Eqs. 1–3, HSC adaptive cosine thresholds Eqs. 4–6, STE-differentiable compression under L_seg + λ L_comp Eq. 7) is an architectural proposal whose parameters are optimized end-to-end on referring data (MeViS + Ref-YouTube-VOS) and then evaluated zero-shot on held-out reasoning benchmarks (ReasonVOS, ReVOS, Ref-DAVIS17). The reported 85 % token reduction and J&F numbers are measured outcomes of that trained policy, not algebraic identities of the inputs. Adaptive thresholds are learned, not fitted to the test metrics themselves. Citations to Mamba, VideoMamba, BIMBA, etc. supply known linear-recurrence properties; none is a load-bearing uniqueness theorem authored by the present team that forbids alternatives. Minor author-overlap references (e.g., SAM2 evaluations) are peripheral. No equation reduces a claimed prediction to a fitted constant by construction, and evaluation uses external public splits never seen in training. Hence the derivation is self-contained against external benchmarks.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The paper is an empirical systems contribution. Its load-bearing content is the architectural ordering (enrich then compress) and the learned adaptive thresholds; almost everything else is standard SSM, CLIP, LLaVA and SAM2 machinery. Free parameters are ordinary training and EMA constants; axioms are domain assumptions about selectivity and causality; invented entities are the two named modules.

free parameters (5)
  • EMA momentum α = 0.1
    Fixed at 0.1 for online mean/variance of similarity scores (Eq. 5); controls how quickly adaptive thresholds adapt.
  • compression loss weight λ = 0.001
    Balances segmentation loss against token-count regularizer (Eq. 7); set to 0.001.
  • threshold offset k (via MLP) = learned
    Learned scalar that sets τ_t = μ_t + k σ_t; directly determines which frames/patches survive.
  • LoRA rank / α = 8 / 16
    Rank-8, α=16 applied only to attention projections of LLaVA-7B.
  • learning rate and schedule = 3e-4
    3e-4 AdamW with 100-step warmup, 3000 iterations, batch size 2.
assumptions (4)
  • domain assumption Selective SSM recurrence produces near-identical enriched states for temporally redundant content, yielding a usable redundancy signal at O(T) cost.
    Stated in introduction and §3.2; underpins the entire enrich-before-compress argument.
  • domain assumption Spatial relationships are non-causal while temporal evolution is strictly causal, justifying decoupled bidirectional-spatial / causal-temporal scans.
    Explicit design premise in §3.1–3.2 that enables streaming.
  • standard math Straight-through estimator allows segmentation gradients to update discrete compression thresholds.
    Invoked in §3.4 citing Bengio et al.; standard technique.
  • domain assumption Frozen CLIP ViT-L/14 features plus LoRA-tuned LLaVA-7B and SAM2 decoder form a sufficient backbone for reasoning segmentation.
    Implementation §4.1; all comparisons rest on this stack.
invented entities (2)
  • State-informed Spatiotemporal Aggregator (SSA)
    purpose: Produces the enriched feature space F_ST in which redundancy becomes measurable before any tokens are dropped.
    Named module introduced in §3.2; no independent existence outside the paper.
  • Hierarchical State-adaptive Compression (HSC)
    purpose: Performs temporal-then-spatial reduction with content-adaptive thresholds driven by the segmentation objective.
    Named module introduced in §3.3; the adaptive policy is learned inside this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation." pith.science (2026). https://pith.science/paper/MC7XFU7C

@misc{pith2026260702922,
  author       = {Pith},
  title        = {Pith review of: STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MC7XFU7C}},
  note         = {Machine review of arXiv:2607.02922}
}
abstract

Video reasoning segmentation demands pixel-accurate object tracking across hundreds of frames under complex natural language queries, producing dense spatiotemporal tokens whose quadratic self-attention cost makes long-video processing prohibitive. Existing methods address this through token compression, yet typically operate on encoder features lacking temporal context, constraining selection before content redundancy can be reliably assessed. Informed compression requires contextual awareness, but acquiring that awareness at full resolution incurs the same quadratic cost compression aims to reduce. State-space models resolve this constraint, as their linear recurrence selectively conditions each token on temporal context at $\mathcal{O}(T)$ cost, producing representations where content redundancy becomes assessable. Building on this, Selective SpatioTemporal Aggregation and Compression (STAC) enriches features via decoupled bidirectional spatial and causal temporal scanning, leveraging recurrence-derived redundancy for hierarchical compression with adaptive thresholds optimised with segmentation objective. STAC achieves 85% token reduction and 1.8$\times$ speedup while surpassing compression-free baselines on reasoning segmentation benchmarks in a zero-shot streaming-compatible setting. Code is available \href{https://github.com/MCG-NKU/nku-video}{here}.

Figures

Figures reproduced from arXiv: 2607.02922 by the authors.

Figure 1
Figure 1. Our architecture introduces a generic spatiotemporal summarizer, which allows for temporal selection and spatiotemporal compression to computation while improving overall performance of the model. with frames {I1, . . . , IT } and text query q, we generate segmentation masks M = {m1, . . . , mT } where mt ∈ {0, 1} H×W . Following standard protocols [5,47], we encode frames through frozen CLIP ViT-L/14 [60] to produc… view at source ↗
Figure 2
Figure 2. Comparison of scanning strategies. Current methods (left) flatten videos into long T×H×W sequences requiring bidirectional processing over the entire video, and use fixed pooling [35, 43] or learnable parameter extraction [32] for compression. Our method (right) decouples bidirectional spatial scanning from causal temporal scan￾ning (spatially parallelized), enabling streaming-compatible content-adaptive compres￾sio… view at source ↗
Figure 3
Figure 3. Performance evaluation across compression methods. STAC maintains efficiency (main) and consistent performance across contexts (inset). occlusion. Most notably, despite training exclusively on referring segmentation data, STAC transfers effectively to reasoning benchmarks, outperforming the strongest compression-free baseline on ReasonVOS [6] in a zero-shot setting. This suggests that informed compression does not m… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance on Segmentation with reasoning shows that our method performs aggressive yet adaptive compression. While the RED areas indicate compressed frames or patches, the use of selective aggregation propagates all the important information within those patches with…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

86 extracted references · 25 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    arXiv preprint arXiv:1412.69801412(6) (2014)

    Adam, K.D.B.J., et al.: A method for stochastic optimization. arXiv preprint arXiv:1412.69801412(6) (2014)

    arXiv 2014

  3. [3]

    Machine Intelligence Research (2026)

    An, Z., Li, Z., Ye, M., Qiao, F., Li, J., Wu, Z., Thengane, V., Li, C., Li, L., Gool, L.V., Sun, G., Belongie, S.: Video understanding: From geometry and semantics to unified models. Machine Intelligence Research (2026)

    2026

  4. [4]

    Machine Intelligence Research (2026)

    Ariff, S.H.S., Liu, Y., Sun, G., Yang, J., Ding, H., Geng, X., Jiang, X.: Evaluating sam2 for video semantic segmentation. Machine Intelligence Research (2026)

    2026

  5. [5]

    arXiv preprint arXiv:2308.12966 (2023)

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023)

    arXiv 2023

  6. [6]

    In: NeurIPS

    Bai, Z., He, T., Mei, H., Wang, P., Gao, Z., Chen, J., Zhang, Z., Shou, M.Z.: One token to seg them all: Language instructed reasoning segmentation in videos. In: NeurIPS. pp. 6833–6859 (2024)

    2024

  7. [7]

    arXiv preprint arXiv:1308.3432 (2013)

    Bengio, Y., Léonard, N., Courville, A.: Estimating or propagating gradi- ents through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013)

    arXiv 2013

  8. [8]

    arXiv preprint arXiv:2210.09461 (2022) 16 Hesham et al

    Bolya, D., Fu, C.Y., Dai, X., Zhang, P., Feichtenhofer, C., Hoffman, J.: Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461 (2022) 16 Hesham et al

    arXiv 2022

Show all 86 references
  1. [9]

    In: CVPR

    Botach, A., Zheltonozhskii, E., Baskin, C.: End-to-end referring video object seg- mentation with multimodal transformers. In: CVPR. pp. 4985–4995 (2022)

    2022

  2. [10]

    arXiv preprint arXiv:2403.09626 (2024)

    Chen, G., Huang, Y., Xu, J., Pei, B., Chen, Z., Li, Z., Wang, J., Li, K., Lu, T., Wang, L.: Video mamba suite: State space model as a versatile alternative for video understanding. arXiv preprint arXiv:2403.09626 (2024)

    2024 arXiv

  3. [11]

    In: ECCV

    Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., Chang, B.: An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In: ECCV. pp. 19–35. Springer (2024)

    2024

  4. [12]

    arXiv preprint arXiv:2408.10188 (2024)

    Chen, Y., Xue, F., Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z., et al.: Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188 (2024)

    2024 arXiv

  5. [13]

    arXiv preprint arXiv:2312.14238 (2023)

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Muyan, Z., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238 (2023)

    2023 arXiv

  6. [14]

    arXiv preprint arXiv:2406.07476 (2024)

    Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., et al.: Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 (2024)

    2024 arXiv

  7. [15]

    arXiv preprint arXiv:2405.21060 (2024)

    Dao, T., Gu, A.: Transformers are ssms: Generalized models and efficient algo- rithms through structured state space duality. arXiv preprint arXiv:2405.21060 (2024)

    2024 arXiv

  8. [16]

    In: ICCV

    Ding, H., Liu, C., He, S., Jiang, X., Loy, C.C.: Mevis: A large-scale benchmark for video segmentation with motion expressions. In: ICCV. pp. 2694–2703 (2023)

    2023

  9. [17]

    In: CVPR

    Ding, Z., Hui, T., Huang, J., Wei, X., Han, J., Liu, S.: Language-bridged spatial- temporal interaction for referring video object segmentation. In: CVPR. pp. 4964– 4973 (2022)

    2022

  10. [18]

    arXiv e-prints pp

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arXiv e-prints pp. arXiv–2407 (2024)

    2024

  11. [19]

    Machine Intelligence Research (2026)

    Feng, Y., Yan, Z., Jia, Y., Chen, E.Q., Qin, J.: Training-free dense video captioning with large-scale pre-trained models. Machine Intelligence Research (2026)

    2026

  12. [20]

    In: CVPR

    Gavrilyuk, K., Ghodrati, A., Li, Z., Snoek, C.G.: Actor and action video segmen- tation from a sentence. In: CVPR. pp. 5958–5966 (2018)

    2018

  13. [21]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Gong, S., Zhuge, Y., Zhang, L., Yang, Z., Zhang, P., Lu, H.: The devil is in tem- poral token: High quality video reasoning segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 29183– 29192 (2025)

    2025

  14. [22]

    In: First Conference on Language Modeling (2024)

    Gu, A., Dao, T.: Mamba: Linear-time sequence modeling with selective state spaces. In: First Conference on Language Modeling (2024)

    2024

  15. [23]

    In: NeurIPS

    Gu, A., Dao, T., Ermon, S., Rudra, A., Ré, C.: Hippo: Recurrent memory with optimal polynomial projections. In: NeurIPS. vol. 33 (2020)

    2020

  16. [24]

    arXiv preprint arXiv:2111.00396 (2021)

    Gu, A., Goel, K., Ré, C.: Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 (2021)

    2021 arXiv

  17. [25]

    In: ICLR (2022)

    Gu, A., Goel, K., Ré, C.: Efficiently modeling long sequences with structured state spaces. In: ICLR (2022)

    2022

  18. [26]

    In: NeurIPS

    Gu, A., Gupta, A., Goel, K., Ré, C.: On the parameterization and initialization of diagonal state space models. In: NeurIPS. vol. 35 (2022)

    2022

  19. [27]

    In: NeurIPS

    Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., Ré, C.: Combining recurrent,convolutional,andcontinuous-timemodelswithlinearstate-spacelayers. In: NeurIPS. vol. 34 (2021)

    2021

  20. [28]

    Machine Intelligence Research (2026) STAC 17

    Han, C., Fan, J., Wu, N., Dai, J., Bao, H., Lu, X.: Object-centric video prediction with mask-guided spatiotemporal diffusion. Machine Intelligence Research (2026) STAC 17

    2026

  21. [29]

    In: CVPR

    He, S., Ding, H.: Decoupling static and hierarchical motion perception for referring video segmentation. In: CVPR. pp. 13332–13341 (2024)

    2024

  22. [30]

    In: ICLR (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: ICLR (2022)

    2022

  23. [31]

    In: CVPR

    Hu, K., Gao, F., Nie, X., Zhou, P., Tran, S., Neiman, T., Wang, L., Shah, M., Hamid, R., Yin, B., et al.: M-llm based video frame selection for efficient video understanding. In: CVPR. pp. 13702–13712 (2025)

    2025

  24. [32]

    In: CVPR

    Islam, M.M., Nagarajan, T., Wang, H., Bertasius, G., Torresani, L.: Bimba: Selective-scan compression for long-range video question answering. In: CVPR. pp. 29096–29107 (2025)

    2025

  25. [33]

    Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., Carreira, J.: Per- ceiver:Generalperceptionwithiterativeattention.In:Int.Conf.MachineLearning. pp. 4651–4664. PMLR (2021)

    2021

  26. [34]

    arXiv:2401.04088 (2024)

    Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., Casas, D.d.l., Hanna, E.B., Bressand, F., et al.: Mixtral of experts. arXiv:2401.04088 (2024)

    2024 arXiv

  27. [35]

    arXiv preprint arXiv:2503.04130 (2025)

    Jiang, J., Li, X., Liu, Z., Li, M., Chen, G., Li, Z., Huang, D.A., Liu, G., Yu, Z., Keutzer, K., et al.: Token-efficient long video understanding for multimodal llms. arXiv preprint arXiv:2503.04130 (2025)

    2025

  28. [36]

    Journal of Basic Engineering82(1), 35–45 (1960)

    Kalman, R.E.: A new approach to linear filtering and prediction problems. Journal of Basic Engineering82(1), 35–45 (1960)

    1960

  29. [37]

    Computational Visual Media11(3), 655–667 (2025)

    Karacan, L., Sarıgül, M.: Full-frame video stabilization via spatiotemporal trans- formers. Computational Visual Media11(3), 655–667 (2025)

    2025

  30. [38]

    In: Asian conference on computer vision

    Khoreva, A., Rohrbach, A., Schiele, B.: Video object segmentation with lan- guage referring expressions. In: Asian conference on computer vision. pp. 123–141. Springer (2018)

    2018

  31. [39]

    In: ICCV

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: ICCV. pp. 4015–4026 (2023)

    2023

  32. [40]

    In: ICCV

    Korbar, B., Tran, D., Torresani, L.: Scsampler: Sampling salient clips from video for efficient action recognition. In: ICCV. pp. 6232–6242 (2019)

    2019

  33. [41]

    In: CVPR

    Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., Jia, J.: Lisa: Reasoning segmentation via large language model. In: CVPR. pp. 9579–9589 (2024)

    2024

  34. [42]

    arXiv preprint arXiv:2408.03326 (2024)

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., Li, C.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

    2024 arXiv

  35. [43]

    In: ECCV

    Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., Qiao, Y.: Videomamba: State space model for efficient video understanding. In: ECCV. pp. 237–255. Springer (2024)

    2024

  36. [44]

    Li, Y., Wang, C., Jia, J.: Llama-vid: An image is worth 2 tokens in large lan- guagemodels.In:EuropeanConferenceonComputerVision.pp.323–340.Springer (2024)

    2024

  37. [45]

    arXiv preprint arXiv:2311.10122 (2023)

    Lin, B., Zhu, B., Ye, Y., Ning, M., Jin, P., Yuan, L.: Video-llava: Learn- ing united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 (2023)

    2023 arXiv

  38. [46]

    In: CVPR

    Lin, L., Yu, X., Pang, Z., Wang, Y.X.: Glus: Global-local reasoning unified into a single large language model for video segmentation. In: CVPR. pp. 8658–8667 (2025)

    2025

  39. [47]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Lee, Y.J.: LLaVA-NeXT: Im- proved reasoning, OCR, and world knowledge (January 2024),https://llava- vl.github.io/blog/2024-01-30-llava-next/, (Accessed: 2026-06-25) 18 Hesham et al

    2024

  40. [48]

    In: NeurIPS

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS. vol. 36, pp. 34892–34916 (2023)

    2023

  41. [49]

    NeurIPS37, 103031–103063 (2024)

    Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., Jiao, J., Liu, Y.: Vmamba: Visual state space model. NeurIPS37, 103031–103063 (2024)

    2024

  42. [50]

    Machine Intelligence Research21(4), 670–683 (2024)

    Liu, Y., Wu, Y.H., Sun, G., Zhang, L., Chhatkuli, A., Van Gool, L.: Vision trans- formers with hierarchical attention. Machine Intelligence Research21(4), 670–683 (2024)

    2024

  43. [51]

    arXiv e-prints pp

    Lu, H., Salah, A.A., Poppe, R.: Videomambapro: A leap forward for mamba in video understanding. arXiv e-prints pp. arXiv–2406 (2024)

    2024

  44. [52]

    In: NeurIPS

    Luo, Z., Xiao, Y., Liu, Y., Li, S., Wang, Y., Tang, Y., Li, X., Yang, Y.: Soc: Semantic-assisted object cluster for referring video object segmentation. In: NeurIPS. vol. 36, pp. 26425–26437 (2023)

    2023

  45. [53]

    In: Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision

    Miao, B., Bennamoun, M., Gao, Y., Mian, A.: Spectrum-guided multi-granularity referring video object segmentation. In: Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision. pp. 920–930 (2023)

    2023

  46. [54]

    In: CVPR

    Munasinghe, S., Gani, H., Zhu, W., Cao, J., Xing, E., Khan, F.S., Khan, S.: Videoglamm: A large multimodal model for pixel-level visual grounding in videos. In: CVPR. pp. 19036–19046 (2025)

    2025

  47. [55]

    Computational Visual Media12(1), 71–84 (2026)

    Ning, M., Zhu, B., Xie, Y., Lin, B., Cui, J., Yuan, L., Chen, D., Yuan, L.: Video- bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. Computational Visual Media12(1), 71–84 (2026)

    2026

  48. [56]

    In: NeurIPS

    Pan, B., Panda, R., Jiang, Y., Wang, Z., Feris, R., Oliva, A.: Ia-red 2: Interpretability-aware redundancy reduction for vision transformers. In: NeurIPS. vol. 34, pp. 24898–24911 (2021)

    2021

  49. [57]

    In: ECCV

    Park, J., Kim, H.S., Ko, K., Kim, M., Kim, C.: Videomamba: Spatio-temporal selective state space model. In: ECCV. pp. 1–18. Springer (2024)

    2024

  50. [58]

    In: AAAI

    Pei, X., Huang, T., Xu, C.: Efficientvmamba: Atrous selective scan for light weight visual mamba. In: AAAI. pp. 6443–6451 (2025)

    2025

  51. [59]

    arXiv preprint arXiv:1704.00675 (2017)

    Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., Van Gool, L.: The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675 (2017)

    2017 arXiv

  52. [60]

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: Int. Conf. Machine Learning. pp. 8748–8763. PmLR (2021)

    2021

  53. [61]

    In: NeurIPS

    Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., Hsieh, C.J.: Dynamicvit: Efficient vision transformers with dynamic token sparsification. In: NeurIPS. vol. 34, pp. 13937–13949 (2021)

    2021

  54. [62]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R.M., Xing, E., Yang, M.H., Khan, F.S.: Glamm: Pixel grounding large multimodal model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13009–13018 (2024)

    2024

  55. [63]

    arXiv preprint arXiv:2408.00714 (2024)

    Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)

    2024 arXiv

  56. [64]

    In: ECCV

    Seo, S., Lee, J.Y., Han, B.: Urvos: Unified referring video object segmentation network with a large-scale benchmark. In: ECCV. pp. 208–223. Springer (2020)

    2020

  57. [65]

    arXiv preprint arXiv:2409.01156 (2024) STAC 19

    Shen, L., Hao, T., He, T., Zhao, S., Zhang, Y., Liu, P., Bao, Y., Ding, G.: Tempme: Video temporal token merging for efficient text-video retrieval. arXiv preprint arXiv:2409.01156 (2024) STAC 19

    2024 arXiv

  58. [66]

    arXiv preprint arXiv:2410.17434 (2024)

    Shen, X., Xiong, Y., Zhao, C., Wu, L., Chen, J., Zhu, C., Liu, Z., Xiao, F., Varadarajan, B., Bordes, F., et al.: Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434 (2024)

    2024 arXiv

  59. [67]

    In: CVPR

    Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., et al.: Moviechat: From dense token to sparse memory for long video understanding. In: CVPR. pp. 18221–18232 (2024)

    2024

  60. [68]

    arXiv preprint arXiv:2508.04369 (2025)

    Tang, C., Han, Z., Sun, H., Zhou, S., Zhang, X., Wei, X., Yuan, Y., Xu, J., Sun, H.: Tspo: Temporal sampling policy optimization for long-form video language understanding. arXiv preprint arXiv:2508.04369 (2025)

    2025

  61. [69]

    In: ICCV

    Tang, J., Zheng, G., Yang, S.: Temporal collection and distribution for referring video object segmentation. In: ICCV. pp. 15466–15476 (2023)

    2023

  62. [70]

    arXiv preprint arXiv:2312.11805 (2023)

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  63. [71]

    arXiv preprint arXiv:2302.13971 (2023)

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  64. [72]

    arXiv:2307.09288 (2023)

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 (2023)

    2023 arXiv

  65. [73]

    In: European Conference on Computer Vision

    Wang, Y., Li, K., Li, X., Yu, J., He, Y., Chen, G., Pei, B., Zheng, R., Wang, Z., Shi, Y., et al.: Internvideo2: Scaling foundation models for multimodal video un- derstanding. In: European Conference on Computer Vision. pp. 396–416. Springer (2024)

    2024

  66. [74]

    Machine Intelligence Research (2026)

    Wang, Z., Shao, D., Zhang, L., Zhang, Z., Wang, B.: SAMDistill: SAM-based spatial-temporal distillation for robust 3d object detection. Machine Intelligence Research (2026)

    2026

  67. [75]

    In: ICCV

    Wu, D., Wang, T., Zhang, Y., Zhang, X., Shen, J.: Onlinerefer: A simple online baseline for referring video object segmentation. In: ICCV. pp. 2761–2770 (2023)

    2023

  68. [76]

    In: CVPR

    Wu, J., Jiang, Y., Sun, P., Yuan, Z., Luo, P.: Language as queries for referring video object segmentation. In: CVPR. pp. 4974–4984 (2022)

    2022

  69. [77]

    arXiv preprint arXiv:1809.03327 (2018)

    Xu, N., Yang, L., Fan, Y., Yue, D., Liang, Y., Yang, J., Huang, T.: Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327 (2018)

    2018 arXiv

  70. [78]

    In: ECCV

    Yan, C., Wang, H., Yan, S., Jiang, X., Hu, Y., Kang, G., Xie, W., Gavves, E.: Visa: Reasoning video object segmentation via large language models. In: ECCV. pp. 98–115. Springer (2024)

    2024

  71. [79]

    arXiv preprint arXiv:2401.14168 (2024)

    Yang, Y., Xing, Z., Yu, L., Huang, C., Fu, H., Zhu, L.: Vivim: A video vision mamba for medical video segmentation. arXiv preprint arXiv:2401.14168 (2024)

    2024 arXiv

  72. [80]

    In: CVPR

    Ye, X., Gan, Y., Ge, Y., Zhang, X.P., Tang, Y.: Atp-llava: Adaptive token pruning for large vision language models. In: CVPR. pp. 24972–24982 (2025)

    2025

  73. [81]

    arXiv preprint arXiv:2501.04001 (2025)

    Yuan, H., Li, X., Zhang, T., Huang, Z., Xu, S., Ji, S., Tong, Y., Qi, L., Feng, J., Yang, M.H.: Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001 (2025)

    2025 arXiv

  74. [82]

    In: ICCV

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. In: ICCV. pp. 11975–11986 (2023)

    2023

  75. [83]

    arXiv preprint arXiv:2306.02858 (2023)

    Zhang, H., Li, X., Bing, L.: Video-llama: An instruction-tuned audio-visual lan- guage model for video understanding. arXiv preprint arXiv:2306.02858 (2023)

    2023 arXiv

  76. [84]

    In: AAAI

    Zhao, H., Zhang, M., Zhao, W., Ding, P., Huang, S., Wang, D.: Cobra: Extending mamba to multi-modal large language model for efficient inference. In: AAAI. pp. 10421–10429 (2025) 20 Hesham et al

    2025

  77. [85]

    arXiv preprint arXiv:2312.17448 (2023)

    Zhu, J., Cheng, Z.Q., He, J.Y., Li, C., Luo, B., Lu, H., Geng, Y., Xie, X.: Tracking with human-intent reasoning. arXiv preprint arXiv:2312.17448 (2023)

    2023 arXiv

  78. [86]

    Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., Wang, X.: Vision mamba: Efficient visual representation learning with bidirectional state space model. In: Int. Conf. Machine Learning. pp. 62429–62442 (2024)

    2024

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.