Pith. sign in

REVIEW 4 major objections 6 minor 54 references

Propagating the SSM hidden state across video chunks reproduces the exact full-video state, so dense captioning can run online with far less memory.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A state-space model with transfer state processes videos chunk by chunk, carries the hidden state forward, and performs online dense video captioning with the same state as full-sequence processing.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A useful online-architecture recipe for dense video captioning, with an over-sold equivalence claim that the paper's own Table 7 contradicts; the empirical results are interesting but need sharper reporting. the 4 major comments →

arxiv 2509.03426 v1 pith:BA2OHNZ7 submitted 2025-09-03 cs.CV

Time-Scaling State-Space Models for Dense Video Captioning

AJ Piergiovanni , Ganesh Satish Mallya , Dahun Kim , Anelia Angelova This is my paper
classification cs.CV
keywords state-space modelsdense video captioningonline video understandinglong videotransfer stateevent localizationstreaming videoS4D
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a linear State-Space Model can be split into video chunks without losing the exact state of the full sequence. The proposed Transfer State mechanism computes the hidden state at the end of each chunk, feeds it into the next chunk as the initial state, and the paper proves that this yields the same hidden state as processing the entire video in one pass. If that proof holds, long-video dense captioning and event localization become online tasks: captions and boundaries can be emitted while the video streams in, memory use is bounded by the current chunk rather than the whole video, and very long or even unbounded videos can be processed. The authors demonstrate gains in captioning quality over global models and report a 7x reduction in FLOPs relative to a recent streaming dense-captioning baseline.

Core claim

The central claim is that the per-segment state update h = Khidden * x + A^(L-1) h0, where Khidden = [B, AB, ..., A^(L-1)B], reproduces the full-sequence hidden state after every segment. The paper's proof unrolls the linear recurrence, splits the input into two parts, and shows that the final hidden state of the two-step computation is exactly the hidden state obtained by feeding the whole signal at once; by induction this holds for any number of segments. Consequently an SSM can be run online: process a snippet by the efficient convolutional form, extract the state, discard the snippet's activations, and continue with the next snippet. Applied to dense video captioning, local text decoders

What carries the argument

The transfer state is the hidden state of the S4D-style linear recurrence, updated in closed form per segment as h_segment = Khidden * x_segment + A^(L-1) h_prev. It compresses all prior video context into a fixed-size vector that is carried across the segment boundary. The load-bearing identity is Eq. 12: when the previous segment's contribution A^(L-M) h_M is unrolled, it expands exactly into the tail terms of the full-sequence convolution, proving that for a linear time-invariant SSM with shared parameters, chunking is lossless. This identity is what converts the memory-hungry full-sequence convolution into an exact streaming computation.

Load-bearing premise

The proof assumes the SSM is a fixed linear recurrence with identical parameters A, B, C, D in every segment and that the complete hidden state of every layer is carried forward, so if the implementation resets the state, reinitializes per-segment modules, or uses input-dependent gating, the chunked computation is no longer exactly the full-sequence computation.

What would settle it

Run the trained STS model twice on the same video, once in chunked mode with transfer state and once as a single full-sequence pass with the same weights, then compare the hidden states at the segment boundaries. If the states differ by more than floating-point tolerance, the claimed exact equivalence does not hold for the implementation as built.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Dense video captioning can be performed genuinely online, since each local decoder holds the complete transfer state of all prior context and does not need any future frames.
  • Inference memory no longer scales with video length; only the current snippet, its activations, and the state vector must reside in memory.
  • Increasing the number of frames per clip improves captioning and localization metrics up to 256 frames, indicating the model continues to time-scale rather than saturate.
  • At matched vision-only inputs, the online STS model beats global models that process the entire video at once on several captioning metrics.
  • The same chunked state mechanism applies, in principle, to arbitrarily long or streaming video, since the video is processed as a sequence of independent snippets with a carry-in state.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The equivalence result is not specific to video: any linear time-invariant SSM on audio, text, or sensor data can be converted into an exact online model by the same per-segment state convolution, provided parameters are shared across chunks.
  • For input-dependent or gated SSM variants, A, B, C, D change per timestep, so Eq. 12 no longer holds exactly; a faithful transfer-state version of such models would need an approximate or alternative identity.
  • The ablation finding that the last token per segment captures almost all the benefit of all tokens hints that the transfer state compels per-snippet compression; this suggests a testable extension where only the segment state is fed to downstream tasks, such as video question answering or moment retrieval.
  • A sharper efficiency claim would compare wall-clock latency on identical hardware for chunked vs. full-sequence inference at matched accuracy, since FLOPs counts alone do not capture memory traffic or scheduling overhead.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes State-Space Models with Transfer State (STS), a chunked/online formulation of S4D for dense video captioning. The input video is divided into snippets; each snippet is processed by an SSM whose hidden state is carried forward and used as the initial state for the next snippet. Section 3.3 gives an algebraic proof that, for a single linear time-invariant recurrence, this chunked computation produces the same final hidden state as processing the full sequence. The model is evaluated on VITT, YouCook2, and ActivityNet for dense captioning and event localization, reporting strong results and a 7x FLOP reduction relative to a prior streaming method. The paper frames STS as enabling online caption generation and scaling SSMs to longer videos.

Significance. If the equivalence in Sec. 3.3 extends to the actual deployed architecture, STS is a useful and conceptually clean way to reduce memory and enable online processing for SSM-based video models. The empirical results are competitive, particularly on SODA/METEOR, and the paper demonstrates scaling to 256 frames, which is beyond prior SSM-based video work. The proof is simple and definitional; no machine-checked proofs or code are provided. However, the exact-equivalence claim is currently over-stated: the proof addresses only a single linear time-invariant recurrence, while the evaluated model is a deep stack of S4D layers with nonlinearities and auxiliary components. The paper's own Table 7 shows that the online SSM and STS do not produce identical outputs, so the central theoretical claim is not established for the model actually used. The efficiency claim also lacks a transparent computation protocol. These issues are addressable but require substantive revision.

major comments (4)
  1. [Sec. 3.3, Eq. (12)] The equivalence proof assumes a single linear time-invariant recurrence with fixed A,B,C,D and no nonlinearities, normalization, or residual connections. The model described in Sec. 3.4 and Figure 2 is a stack of S4D layers with nonlinearities and is combined with a ViT encoder, a bottleneck Transformer, and a text decoder. For Eq. (12) to describe the evaluated system, the paper must specify (i) that A,B,C,D are tied across all segments, (ii) that the full hidden state of every S4D layer is propagated at segment boundaries, not a single summary state, and (iii) that any normalization aggregating over time is absent or uses global statistics. None of these is specified. As written, the proof does not cover the architecture that is actually trained and evaluated.
  2. [Sec. 4, Table 7] Table 7 compares the online SSM and STS and reports close but not identical numbers: CIDEr 23.8 vs 24.0, mIOU 39.6 vs 39.7, F1 27.8 vs 27.6. The caption says 'As expected they are comparable,' but if Eq. (12) described the evaluated model, the outputs should match up to floating-point nondeterminism in a single run. The authors need to explain this discrepancy: is it due to training stochasticity, different inference code, or an algorithmic mismatch? Without such an explanation, the exact-equivalence claim is not supported by the experiments.
  3. [Sec. 4, Table 4] The abstract and Sec. 1 claim a '7x fewer FLOPs' advantage, but the supporting comparison is a single number against SDC [50], a method with a different architecture, input resolution, and output-token count. No formula, breakdown, or like-for-like computation protocol is provided. Because the model includes a 300M ViT, a bottleneck Transformer, and a text decoder, it is unclear whether the FLOP saving is due to SSM chunking or to other modeling choices. This claim should be substantiated with a transparent FLOP calculation and, ideally, a controlled comparison.
  4. [Sec. 1 and Sec. 3.3] The paper motivates STS by citing Merrill et al. [42] that SSMs cannot sustain their state over long contexts, but the chunked computation described in Eq. (12) preserves exactly the same finite-dimensional linear state as the full-sequence computation; it does not expand the state capacity or change the dynamics. Therefore, STS cannot by itself 'address the main limitation of SSMs' regarding state deterioration. The empirical gains over the global SSM in Table 3 may come from the online/local decoder or from training differences rather than from improved state sustainability. The paper should either reframe this motivation or provide evidence that state sustainability, rather than the decoder, drives the improvement.
minor comments (6)
  1. [Sec. 3.3, Eq. (12)] There appears to be an indexing/exponent error in the last line of Eq. (12): the term for x_1 should be A^{L-1}B x_1, not A^L B x_1, and the term A^{L-M}B x_{L-M+1} is duplicated. Please correct the unrolling.
  2. [Sec. 4, Table 7 caption] The phrase 'As expected they are comparable' is at odds with the exact-equivalence claim in Sec. 3.3. If exact, the results should be identical up to floating-point noise; 'comparable' suggests an approximation.
  3. [Sec. 3.4, Implementation details] The text says '8 segments per video and 128 frames per video' but later ablations use 16-256 frames and the model is described as arbitrarily scalable. Clarify the relationship between the stated configuration and the scalability experiments.
  4. [Sec. 4, Ablations] The sentence 'In the final model, we used all the tokens, corresponding to the last frame of the segment' is confusing: 'all the tokens' of the segment versus tokens of the last frame are different quantities. Please clarify the token selection.
  5. [References] Reference [16] has a garbled author list ('Jiahao Wang Jilan Xu Yifei Huang ...') and should be formatted properly.
  6. [Sec. 3, Eqs. (1)-(4)] The same symbols A,B,C,D are used for continuous-time and discretized matrices. The discretized versions should be denoted differently (e.g., A_bar, B_bar) to avoid confusion, especially because the proof in Sec. 3.3 uses the discretized recurrence.

Circularity Check

0 steps flagged

No significant circularity: the STS equivalence proof is a direct algebraic unrolling of the SSM recurrence, and all empirical claims are benchmarked externally.

full rationale

The central derivation in Sec. 3.2-3.3 is self-contained. Equation 12 follows by unrolling the recurrence h_k = A h_{k-1} + B x_k and substituting the segment-level state h_M into the next segment; this is a mathematical identity for a fixed linear time-invariant SSM, not a fitted prediction and not a result that presupposes its own conclusion. The claimed equivalence is definitional in the algebraic sense, but it is not circular with respect to experimental data: it involves no fitted parameters and is not validated by the benchmark numbers. The dense captioning results (Tables 2, 3, 5, 6) are comparisons against external datasets and prior methods, and the efficiency claim (Table 4) is a FLOPs calculation rather than a prediction derived from a fitted input. Self-citations (e.g., Mirasol3B for the bottleneck Transformer, Vid2Seq for decoder design) are architectural borrowings and are not load-bearing for the equivalence proof, nor do they invoke any uniqueness theorem. One potential overstatement is worth noting but is not circularity: Table 7 reports online SSM and STS values that are close but not identical (e.g., CIDEr 23.8 vs 24.0, F1 27.8 vs 27.6), suggesting the exact-equivalence proof may not fully cover the deployed multi-layer S4D stack with normalization and residual connections. The paper itself only says the two are 'comparable' in Table 7, and the proof remains valid for the stated single-recurrence setting. Thus no step reduces to its inputs by construction, and the correct circularity score is 0.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The central mathematical claim has no fitted constants beyond standard learned parameters; the listed free parameters are architecture choices. The proof relies on the linearity and time-invariance of the SSM recurrence and on parameter sharing across segments. No new physical or conceptual entities are introduced; 'Transfer State' is a name for the existing SSM hidden state.

free parameters (2)
  • Segment and frame configuration = 8 segments, 128 frames (up to 256); state and bottleneck size 16 in final model
    Chosen by hand for the final system. Frame-count ablations are shown, but segment length, state size, and bottleneck size have no sensitivity analysis.
  • Timestamp buckets = 32
    Timestamps are discretized into 32 buckets after normalizing by video length; the number of buckets is a design choice not justified by ablation.
axioms (3)
  • standard math The S4D layer is a linear time-invariant recurrence h_k = A h_{k-1} + B x_k with fixed A,B,C,D (Eqs. 3, 9)
    The transfer-state equivalence in Sec. 3.3 unrolls this recurrence; if the deployed layer uses selective or input-dependent state updates, the proof does not apply.
  • domain assumption SSM parameters are shared across all segments and the full hidden state is passed intact
    Eq. 12 requires the same A,B,C,D over both segments and exact carry-over of h_M. The paper states this at a high level but does not detail weight tying or per-layer state handling.
  • domain assumption The fixed-dimensional hidden state can preserve long-range context
    The paper's motivation claims SSM state deteriorates for long inputs and that STS sustains information; this presumes the state is a meaningful summary, while the cited work [42] shows expressiveness limits of such states.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Time-Scaling State-Space Models for Dense Video Captioning." pith.science (2026). https://pith.science/paper/BA2OHNZ7

@misc{pith2026250903426,
  author       = {Pith},
  title        = {Pith review of: Time-Scaling State-Space Models for Dense Video Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BA2OHNZ7}},
  note         = {Machine review of arXiv:2509.03426}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Dense video captioning is a challenging video understanding task which aims to simultaneously segment the video into a sequence of meaningful consecutive events and to generate detailed captions to accurately describe each event. Existing methods often encounter difficulties when working with the long videos associated with dense video captioning, due to the computational complexity and memory limitations. Furthermore, traditional approaches require the entire video as input, in order to produce an answer, which precludes online processing of the video. We address these challenges by time-scaling State-Space Models (SSMs) to even longer sequences than before. Our approach, State-Space Models with Transfer State, combines both the long-sequence and recurrent properties of SSMs and addresses the main limitation of SSMs which are otherwise not able to sustain their state for very long contexts, effectively scaling SSMs further in time. The proposed model is particularly suitable for generating captions on-the-fly, in an online or streaming manner, without having to wait for the full video to be processed, which is more beneficial in practice. When applied to dense video captioning, our approach scales well with video lengths and uses 7x fewer FLOPs.

Figures

Figures reproduced from arXiv: 2509.03426 by AJ Piergiovanni, Anelia Angelova, Dahun Kim, Ganesh Satish Mallya.

Figure 1
Figure 1. Figure 1: We propose State-Space Models with Transfer State to extend SSMs to tasks such [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: At a high level, visual tokens from frames, belonging to video snippets, are fed to [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 36 canonical work pages · 1 internal anchor

  1. [1]

    Flamingo: a visual language model for few-shot learning,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Binkowsk...

    work page

  2. [2]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6836–6846, 2021

    2021

  3. [3]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summa- rization, pages 65–72, 2005

    work page 2005

  4. [4]

    Is space-time attention all you need for video understanding? 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? 2021

    work page 2021

  5. [5]

    Hierarchical state space models for contin- uous sequence-to-sequence modeling

    Raunaq Bhirangi, Chenyu Wang, Venkatesh Pattabiraman, Carmel Majidi, Abhinav Gupta, Tess Hellebrekers, and Lerrel Pinto. Hierarchical state space models for contin- uous sequence-to-sequence modeling. In arxiv.org/abs/2402.10211, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017

    2017

  7. [7]

    Video mamba suite: State space model as a ver- satile alternative for video understanding

    Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jiahao Wang, Kun- chang Li, Tong Lu, and Limin Wang. Video mamba suite: State space model as a ver- satile alternative for video understanding. In https://arxiv.org/abs/2403.09626, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    PaLI-X: On scaling up a multilingual vision and language model

    Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Xiao Wang Sebastian Goodman, Yi Tay, Daniel Salz Sia- mak Shakeri, Mostafa Dehghani, Mario Lucic, Michael Tschannen, Arsha Nagrani, PIERGIOV ANNI, MALL Y A, KIM, ANGELOV A: TIME-SCALING STA TE-SPACE MODELS11 Hexiang Hu, Mandar Joshi, Bo Pang, Cesl...

    Pith/arXiv arXiv 2023

  9. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  10. [10]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jiten- dra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In arxiv.org/abs/2104.11227, 2021

    Pith/arXiv arXiv 2021

  11. [11]

    Soda: Story oriented dense video captioning evaluation framework

    Soichiro Fujita, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. Soda: Story oriented dense video captioning evaluation framework. In Com- puter Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16, pages 517–531. Springer, 2020

    work page 2020

  12. [12]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In arXiv:2312.00752, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Combining recurrent, convolutional, and continuous-time models with the structured learnable linear state space layer

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christo- pher Re. Combining recurrent, convolutional, and continuous-time models with the structured learnable linear state space layer. 2021

    work page 2021

  14. [14]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In ICLR, 2022

    work page 2022

  15. [15]

    On the parameterization and initialization of diagonal state space models

    Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. 2022

    work page 2022

  16. [16]

    Videollm: Modeling video se- quence with large language models

    Jiahao Wang Jilan Xu Yifei Huang Junting Pan Yi Wang Yali Wang Yu Qiao Tong Lu Guo Chen, Yin-Dong Zheng and Limin Wang. Videollm: Modeling video se- quence with large language models. https://arxiv.org/abs/2305.13292), year=2023

    Pith/arXiv arXiv 2023

  17. [17]

    Towards evaluating the robustness of visual state space models

    Muzammal Naseer Karthik Nandakumar Fahad Shahbaz Khan Salman Khan Hashmat Shadab Malik, Fahad Shamshad. Towards evaluating the robustness of visual state space models. In https://arxiv.org/abs/2406.09407, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Ac- tivitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Ac- tivitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015

    work page 2015

  19. [19]

    Multimodal pretraining for dense video captioning

    Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, and Radu Soricut. Multimodal pretraining for dense video captioning. In AACL-IJCNLP, 2020. 12 PIERGIOV ANNI, MALL Y A, KIM, ANGELOV A: TIME-SCALING STA TE-SPACE MODELS

    work page 2020

  20. [20]

    A better use of audio-visual cues: Dense video cap- tioning with bi-modal transformer

    Vladimir Iashin and Esa Rahtu. A better use of audio-visual cues: Dense video cap- tioning with bi-modal transformer. In BMVC, 2020

    work page 2020

  21. [21]

    Long movie clip classification with state- space video model

    Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state- space video model. In ECCV, 2022

    work page 2022

  22. [22]

    Long movie clip classification with state-space video model

    Md Mohaiminul Islam, Mahmudul Hasan, Kishan Shamsundar Athrey, Tony Braskich, and Gedas Bertasius. Long movie clip classification with state-space video model. In CVPR, 2023

    work page 2023

  23. [23]

    Linderman Jimmy T.H

    Scott W. Linderman Jimmy T.H. Smith, Andrew Warrington. Simplified state space layers for sequence modeling. In https://arxiv.org/abs/2208.04933, 2022

    Pith/arXiv arXiv 2022

  24. [24]

    MaMMUT: A simple architecture for joint learning for multimodal tasks

    Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Andrew Dai Luowei Zhou, Zhifeng Chen, Claire Cui, and Anelia Angelova. MaMMUT: A simple architecture for joint learning for multimodal tasks. In Transac- tions on Machine Learning Research, 2023

    work page 2023

  25. [25]

    Videomamba: State space model for efficient video understanding

    Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. In https://arxiv.org/pdf/2403.06977, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Video swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022

    2022

  27. [27]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022

    work page 2022

  28. [28]

    Downs, Preey Shah, Tri Dao, Stephen A

    Eric Nguyen, Karan Goel, Albert Gu, Gordon W. Downs, Preey Shah, Tri Dao, Stephen A. Baccus, and Christopher Ré. S4ND: Modeling images and videos as mul- tidimensional signals using state spaces. 2022

    work page 2022

  29. [29]

    Ssm meets video diffusion models: Efficient long-term video generation with selective state spaces

    Yuta Oshimaa, Shohei Taniguchia, Masahiro Suzukia, and Yutaka Matsuoa. Ssm meets video diffusion models: Efficient long-term video generation with selective state spaces. In https://arxiv.org/pdf/2403.07711, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    VideoMamba: Spatio-Temporal Selective State Space Model

    Jinyoung Park, Hee-Seon Kim, Kangwook Ko, Minbeom Kim, and Changick Kim. Videomamba: Spatio-temporal selective state space model. arXiv preprint arXiv:2407.08476, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [31]

    Rethinking video vits: Sparse video tubes for joint image and video learning

    AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova. Rethinking video vits: Sparse video tubes for joint image and video learning. CVPR, 2023

    work page 2023

  32. [32]

    Dynamic pretraining of vision-language models

    AJ Piergiovanni, Weicheng Kuo, Wei Li, and Anelia Angelova. Dynamic pretraining of vision-language models. In First workshop on Multimodal Representation Learning, International Conference on Learning Representations (ICLR), 2023

    work page 2023

  33. [33]

    Mirasol3B: A multimodal autoregressive model for time-aligned and con- textual modalities

    AJ Piergiovanni, Isaac Noble, Dahun Kim, Michael Ryoo, Victor Gomes, and Anelia Angelova. Mirasol3B: A multimodal autoregressive model for time-aligned and con- textual modalities. In CVPR, 2024. PIERGIOV ANNI, MALL Y A, KIM, ANGELOV A: TIME-SCALING STA TE-SPACE MODELS13

    work page 2024

  34. [34]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    work page 2024

  35. [35]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017

    work page 2017

  36. [36]

    Cider: Consensus- based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 4566–4575, 2015

    work page 2015

  37. [37]

    Git: A generative image-to-text transformer for vision and language

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022

    Pith/arXiv arXiv 2022

  38. [38]

    Bidirectional attentive fusion with context gating for dense video captioning

    Jingwen Wang, Wenhao Jiang, Lin Ma, Wei Liu, and Yong Xu. Bidirectional attentive fusion with context gating for dense video captioning. In CVPR, 2018

    work page 2018

  39. [39]

    Selective structured state-spaces for long-form video understanding

    Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In CVPR, 2023

    work page 2023

  40. [40]

    Omnivid: A generative framework for universal video understanding

    Junke Wang, Dongdong Chen, Chong Luo, Bo He, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. Omnivid: A generative framework for universal video understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18209–18220, 2024

    work page 2024

  41. [41]

    End- to-end dense video captioning with parallel decoding

    Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End- to-end dense video captioning with parallel decoding. In ICCV, 2021

    work page 2021

  42. [42]

    The illusion of state in state-space models

    Ashish Sabharwal William Merrill, Jackson Petty. The illusion of state in state-space models. In arxiv.org/pdf/2404.08819, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Dibs: Enhancing dense video captioning with unlabeled videos via pseudo boundary enrichment and online refinement

    Hao Wu, Huabin Liu, Yu Qiao, and Xiao Sun. Dibs: Enhancing dense video captioning with unlabeled videos via pseudo boundary enrichment and online refinement. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024

    work page 2024

  44. [44]

    Vid2seq: Large-scale pretraining of a visual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. CVPR, 2023

    work page 2023

  45. [45]

    Hierarchical video-moment retrieval and step-captioning

    Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mohit Bansal. Hierarchical video-moment retrieval and step-captioning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23056–23065, 2023

    work page 2023

  46. [46]

    Merlot: Multimodal neural script knowledge models

    Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. 2021

    work page 2021

  47. [47]

    Unifying event detection and captioning as sequence generation via pre-training

    Qi Zhang, Yuqing Song, and Qin Jin. Unifying event detection and captioning as sequence generation via pre-training. In ECCV, 2022. 14 PIERGIOV ANNI, MALL Y A, KIM, ANGELOV A: TIME-SCALING STA TE-SPACE MODELS

    work page 2022

  48. [48]

    Towards automatic learning of pro- cedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of pro- cedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    work page 2018

  49. [49]

    End- to-end dense video captioning with masked transformer

    Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End- to-end dense video captioning with masked transformer. CVPR, 2018

    work page 2018

  50. [50]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In CVPR, 2024

    work page 2024

  51. [51]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In arxiv.org/pdf/2401.09417, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Thapliyal, William Yang Wang, and Radu Soricut

    Wanrong Zhu, Bo Pang, Ashish V . Thapliyal, William Yang Wang, and Radu Soricut. End-to-end dense video captioning as sequence generation. In COLING, 2022

    work page 2022

  53. [53]

    State space models for event cameras

    Nikola Zubic, Mathias Gehrig, and Davide Scaramuzza. State space models for event cameras. In CVPR, 2024

    work page 2024

  54. [2022]

    URL https://arxiv.org/abs/2204.14198

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.