Pith. sign in

REVIEW 4 major objections 6 minor 59 references

STLight: a Fully Convolutional Approach for Efficient Predictive Learning by Spatio-Temporal joint Processing

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read STLight claims a purely convolutional network can match or beat recurrent and attention-based video predictors on standard benchmarks while using far fewer parameters and FLOPs.

desk verdict Solid efficient fully-conv video predictor with genuine gains; accuracy claims need reining in before publication. read the letter →

arxiv 2411.10198 v1 pith:CV4NI24G submitted 2024-11-15 cs.CV

Andrea Alfarano , Alberto Alfarano , Linda Friso , Andrea Bacciu , Irene Amerini , Fabrizio Silvestri This is my paper · ORCID
classification cs.CV
keywords spatio-temporalpredictivelearningvideopredictionfullyconvolutionalnetworkdepthwiseconvolutionpatchespixelshuffleself-supervisedefficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a purely convolutional network, with no recurrence and no attention, can match or beat the best recurrent and recurrent-free models on standard spatio-temporal prediction benchmarks while using far fewer parameters and FLOPs. The route is to stop encoding frames independently. Instead, STLight stacks all observed frames along the channel dimension and carves them into overlapping spatio-temporal patches with a single convolution, so each patch token carries both where things are and when they happen. A mixer of depthwise convolutions then combines local and distant patch relationships before a parameter-free pixel shuffle rebuilds the predicted frames. If the claim holds, it removes the main argument for recurrent units in video prediction and gives resource-constrained settings a simpler, scalable baseline.

What carries the argument

The spatio-temporal patch: frames stacked in the channel dimension are convolved with overlapping patches, so each output token encodes spatial position and temporal phase together, and a single convolution replaces the separate spatial encoder and temporal translator of standard Spatial-Temporal-Spatial designs. The STLMixer block is a ConvMixer-style unit with two depthwise convolutions, a small kernel for local detail and a dilated larger kernel for distant context, followed by a 1x1 pointwise convolution; repeated blocks with inter-block skip connections let near and far patches exchange information. The decoder's PixelShuffle operation is parameter-free and rearranges patch dimensions back to full resolution before a single 1x1 convolution reassembles the output frames.

What would settle it

Re-train each baseline with its original authors' training protocols and optimal hyperparameters rather than the benchmark defaults; if tuned ConvLSTM, PredRNN, or TAU then beat STLight's reported MSE and SSIM at comparable FLOPs, the claimed state-of-the-art status fails.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in convolutional video prediction is not convolution itself, but the habit of treating frames as separate spatial inputs. STLight interleaves the T observed frames along the channel dimension into a single tensor and applies one convolution to form overlapping patches whose hidden dimension mixes spatial and temporal content. The resulting STLMixer stack alternates a small-kernel depthwise convolution and a dilated larger-kernel depthwise convolution, with a pointwise mixer, to capture both close and distant patch interactions without attention or recurrence. The decoder is a PixelShuffle followed by one 1x1 convolution, so the only learnable layers in the whole model are channel-wise and depth-wise convolutions. On Moving MNIST, TaxiBJ and KTH, the authors report that STLight variants match or beat published state-of-the-art results while using a fraction of the parameters and FLOPs of recurrent baselines, and they report better sample efficiency and cross-dataset generalisation on KITTI-to-Caltech.

Load-bearing premise

The headline comparison rests on the assumption that the public benchmark numbers used for every baseline are faithful, well-tuned implementations of the original methods; the paper itself notes in its appendix that the benchmark suite does not guarantee correctness of implementations, adherence to original training protocols, or optimality of default training parameters.

Editorial extensions

If this is right

  • Recurrent units are not required for state-of-the-art frame prediction; channel-wise interleaving of frames plus a single convolution gives temporal characterisation.
  • Resource-constrained applications can deploy competitive predictors at a fraction of the FLOPs: for example, STLight-S on Moving MNIST uses about 14% of the FLOPs of PredRNN while achieving a lower MSE.
  • Long-sequence prediction, such as KTH 20-frame forecasting, can be done in one fully parallel forward pass rather than recursive rollouts, with STLight-L matching PredRNNv2 using roughly 2% of its FLOPs.
  • Cross-dataset generalisation from KITTI to Caltech improves with STLight scale, with strong results across the 0.1M to 15M parameter range.
  • Training is sample-efficient: STLight reaches lower MSE in fewer epochs than recurrent and recurrent-free baselines under identical schedules.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the public baseline numbers the paper compares against are re-tuned to their original authors' protocols, the size of STLight's advantage may shrink; the paper itself notes that its benchmark suite does not guarantee faithful, optimally tuned baselines.
  • The spatio-temporal patch recipe could transfer to other frame-sequence prediction tasks, such as radar nowcasting or surgical video, where channel-stacking frames is already a common preprocessing step.
  • The claim that wide dilated depthwise convolutions can substitute for attention in this setting suggests a cheaper inductive bias for long-horizon video prediction; a direct test would swap the STLMixer's dilated kernel for an attention block at equal parameter budget.
  • Because the decoder is a parameter-free PixelShuffle plus one 1x1 convolution, the architecture is unusually easy to port to edge and mobile accelerators, an implication the paper gestures at only through FLOP counts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes STLight, a fully convolutional architecture for spatio-temporal predictive learning (STL). It encodes input frame sequences into overlapping spatio-temporal patches using a single convolution, processes these patches with repeated STLMixer blocks that combine local and dilated depthwise convolutions, and decodes via PixelShuffle and a 1x1 convolution. The authors report accuracy and efficiency results on Moving MNIST, TaxiBJ, KTH, and a cross-dataset generalization test (KITTI-to-Caltech), claiming state-of-the-art accuracy with substantially lower parameter counts and FLOPs than recurrent baselines. The paper includes ablations of the encoder, decoder, mixer, skip connections, and hyperparameters, plus an appendix with extended training, an alternative literature comparison, and the full PyTorch implementation.

Significance. If the accuracy claims were fully supported, STLight would be a notable contribution: it demonstrates that a simple, fully convolutional design can be competitive with recurrent models in predictive learning while being markedly more efficient. The efficiency gains are real and well documented in the parameter/FLOP tables, and the public code and appendix implementation strengthen reproducibility. The ablation study and sample-efficiency analysis are useful. However, the central 'state-of-the-art accuracy' claim is not consistently supported by the paper's own results, and the contribution should be reframed as an efficient architecture that is competitive under specific benchmark protocols rather than unconditionally SOTA.

major comments (4)
  1. [Abstract and §4.3] The abstract and contribution list state that STLight 'achieves state-of-the-art performance' and 'outperforming or matching previous methods in terms of accuracy'. This is contradicted by the paper's own extended-training results in Appendix A, Table 6: at 2000 epochs on MMNIST, PredRNN++ attains MSE 14.07 and SSIM 0.9698, both better than STLight-L's MSE 14.77 and SSIM 0.9686. The claim must be qualified to the specific training settings (standard OpenSTL protocol, 200 epochs) and should be framed as an accuracy-efficiency tradeoff, not unconditional SOTA accuracy.
  2. [§4.2.2 and Table 3] The text in §4.2.1 says STLight 'outperforms both recurrent-based and recurrent-free models across all metrics'. This is not accurate: on TaxiBJ, TAU achieves lower MAE (14.93 vs STLight-L's 15.00), and on KTH, PredRNN++ achieves higher PSNR (28.13 vs 27.57) and SSIM (0.9124 vs 0.9113). The wording should be corrected to reflect that STLight is best or near-best on some metrics and datasets but not all.
  3. [Appendix A, Table 6 and surrounding text] The narrative in Appendix A misreports its own data. The text claims that 'our model continues to outperform recurrent architectures like PredRNN++ and MIM in accuracy', but Table 6 shows PredRNN++ with lower MSE (14.07 vs 14.77) and higher SSIM (0.9698 vs 0.9686) than STLight-L. The appendix is meant to corroborate the main-table conclusions, yet it actually demonstrates that the accuracy ordering can invert with training duration. This internal inconsistency is load-bearing for the paper's central claim and must be fixed.
  4. [§4.1 Train-eval settings and Appendix A] The main results in Tables 3, 5, and 6 are single runs with no error bars, while the baselines are taken from the OpenSTL benchmark. The authors themselves acknowledge in Appendix A that OpenSTL 'does not fully guarantee (1) the correctness of the implementations, (2) the adherence to the original training protocols of each baseline, or (3) the optimality of the default standard training parameters'. Given this caveat and the small differences on some metrics (e.g., KTH SSIM 0.9113 vs 0.9124), the term 'state-of-the-art' should be replaced by a more circumspect formulation such as 'competitive under the OpenSTL evaluation protocol', and error bars or multiple-seed results should be provided for the main comparisons.
minor comments (6)
  1. [Table 6] The last row lists 'STLight-L (Ours) 32.9M 32.9M'; the second occurrence is clearly a typo and should be 32.9G for FLOPs.
  2. [Equation (1)] Equation (1) uses an unclear notation for indexing frames; it should be written explicitly as X ∈ R^{T×C×H×W} with a definition that the frames range from t0−T+1 to t0.
  3. [§4.5 and Appendix C] In §4.5, the text refers to 'Figure 7' for the kernel-size study, but the actual figure is in Appendix C; please make the cross-reference explicit.
  4. [Appendix E] The text says 'In Tables 8 and ??' — the placeholder '??' should be replaced with the actual table number.
  5. [Throughout] The phrase 'state-of-the-art' appears in the abstract and contributions without specifying the benchmark or training protocol; please qualify it consistently (e.g., 'under the OpenSTL training settings') to match the actual evidence.
  6. [§1 Contributions] The claim 'For the first time in the STL context, we jointly process temporal and spatial dynamics' may be overstated given earlier joint spatio-temporal processing in 3D convolution and video diffusion models; consider softening the novelty claim or citing comparative prior work.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: STLight is an empirical benchmark comparison against external OpenSTL baselines; no derivation reduces to its own inputs.

full rationale

This is an empirical systems/benchmark paper, not a first-principles derivation, so the standard circularity patterns do not apply. The central claim ('state-of-the-art performance ... significantly improving computational efficiency') is supported by comparisons against OpenSTL baseline numbers, which are external evidence rather than fitted inputs or self-citations; the OpenSTL and TAU authors are not among the present authors. Hyperparameters are selected by grid-search on validation loss, a standard model-selection procedure, and the reported test metrics are not themselves used as fitting targets. The paper's Appendix A admission that OpenSTL does not guarantee correctness, protocol adherence, or optimality of baseline training parameters is a benchmark-fidelity caveat, not circularity: it weakens the strength of the external evidence but does not make the conclusion equivalent to its premises. The extended-training result in Table 6, where PredRNN++ beats STLight-L on MMNIST MSE and SSIM at 2000 epochs, is a robustness or correctness concern for the unqualified SOTA claim, not a circular-structure concern. No step in the paper's reasoning reduces by construction to its own inputs, so the circularity score is 0.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central empirical claim rests on benchmark fidelity assumptions and on the adequacy of grid-searched hyperparameters. There are no invented physical entities. Free parameters are architectural and training choices, each tuned per dataset.

free parameters (6)
  • hidden dimension d = 1000 to 1800 (e.g., 1400 for STLight-33M)
    Controls patch representation width; tuned per model size in Figures 6 and Table 8.
  • mixer depth de = 16
    Number of STLMixer blocks; elbow analysis in Section 4.5 selects de near 16.
  • patch size p = 2 (1 for TaxiBJ)
    Spatial downsampling factor; chosen based on input resolution.
  • overlap O = 0 or 2
    Patch overlap; Section 4.5 shows larger models benefit from O = 2.
  • kernel sizes kT1, kT2 = 3 and 7
    Depthwise kernel sizes for local and dilated stages; tuned in Appendix C.
  • learning rate = 0.003 (MMNIST), 0.003 (TaxiBJ), 0.0005 (KTH), 0.01 (Caltech)
    Per-dataset learning rate from Table 2, validated in Appendix E.
assumptions (3)
  • domain assumption OpenSTL benchmark baselines are correctly implemented and fairly comparable
    The SOTA claim depends on the published OpenSTL results being faithful; the paper itself flags this in Appendix A.
  • domain assumption MSE as the sole training objective is sufficient to achieve competitive SSIM and PSNR
    The model minimizes MSE only, yet claims SOTA on other metrics; no other loss terms are used.
  • ad hoc to paper Grid-search selected hyperparameters on validation generalize to test sets
    Hyperparameters are selected per dataset via grid search, which risks overfitting to validation; standard practice but an assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of STLight: a Fully Convolutional Approach for Efficient Predictive Learning by Spatio-Temporal joint Processing." pith.science (2026). https://pith.science/paper/CV4NI24G

@misc{pith2026241110198,
  author       = {Pith},
  title        = {Pith review of: STLight: a Fully Convolutional Approach for Efficient Predictive Learning by Spatio-Temporal joint Processing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CV4NI24G}},
  note         = {Machine review of arXiv:2411.10198}
}
read the original abstract

Spatio-Temporal predictive Learning is a self-supervised learning paradigm that enables models to identify spatial and temporal patterns by predicting future frames based on past frames. Traditional methods, which use recurrent neural networks to capture temporal patterns, have proven their effectiveness but come with high system complexity and computational demand. Convolutions could offer a more efficient alternative but are limited by their characteristic of treating all previous frames equally, resulting in poor temporal characterization, and by their local receptive field, limiting the capacity to capture distant correlations among frames. In this paper, we propose STLight, a novel method for spatio-temporal learning that relies solely on channel-wise and depth-wise convolutions as learnable layers. STLight overcomes the limitations of traditional convolutional approaches by rearranging spatial and temporal dimensions together, using a single convolution to mix both types of features into a comprehensive spatio-temporal patch representation. This representation is then processed in a purely convolutional framework, capable of focusing simultaneously on the interaction among near and distant patches, and subsequently allowing for efficient reconstruction of the predicted frames. Our architecture achieves state-of-the-art performance on STL benchmarks across different datasets and settings, while significantly improving computational efficiency in terms of parameters and computational FLOPs. The code is publicly available

Figures

Figures reproduced from arXiv: 2411.10198 by the authors.

Figure 1
Figure 1. MSE vs Number of parameters for existing STL models [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. STLight model workflow. We rearrange the input sequence of frames along the channel dimension (1), and through a single [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative results on Moving MNIST and TaxiBJ datasets. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: STLight models trained on KITTI (0.1M-15M parame [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Learning curve comparison be￾tween state-of-the-art methods and ours. 10 20 30 40 50 60 70 21 22 23 24 25 Parameters (in millions) MSE d=1000 d=1200 d=1400 d=1600 d=1800 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 8
Figure 8. Figure 8: Learning curve comparison for three different weights [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Dispersion vs Number of parameters on different decod [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Full STLight implementation [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 39 canonical work pages

  1. [1]

    Improving generalization of transfer learning across do- mains using spatio-temporal features in autonomous driving

    Shivam Akhauri, Laura Zheng, Tom Goldstein, and Ming Lin. Improving generalization of transfer learning across do- mains using spatio-temporal features in autonomous driving. arXiv preprint arXiv:2103.08116, 2021. 1

    arXiv 2021

  2. [2]

    Efficient prediction of human motion for real-time robotics applica- tions with physics-inspired neural networks

    Alessandro Antonucci, Gastone Pietro Rosati Papini, Paolo Bevilacqua, Luigi Palopoli, and Daniele Fontanelli. Efficient prediction of human motion for real-time robotics applica- tions with physics-inspired neural networks. IEEE Access, 10:144–157, 2021. 1

    work page 2021

  3. [3]

    Anticipating many futures: Online human motion predic- tion and generation for human-robot interaction

    Judith B ¨utepage, Hedvig Kjellstr ¨om, and Danica Kragic. Anticipating many futures: Online human motion predic- tion and generation for human-robot interaction. In 2018 IEEE international conference on robotics and automation (ICRA), pages 4563–4570. IEEE, 2018. 1

    work page 2018

  4. [4]

    Mau: A motion- aware unit for video prediction and beyond

    Zheng Chang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, Yan Ye, Xiang Xinguang, and Wen Gao. Mau: A motion- aware unit for video prediction and beyond. Advances in Neural Information Processing Systems , 34:26950–26962,

  5. [5]

    Cyclemlp: A mlp-like architecture for dense prediction

    Shoufa Chen, Enze Xie, Chongjian Ge, Runjian Chen, Ding Liang, and Ping Luo. Cyclemlp: A mlp-like architecture for dense prediction. arXiv preprint arXiv:2107.10224, 2021. 2, 3

    arXiv 2021

  6. [6]

    Spatio-temporal image representation and deep- learning-based decision framework for automated vehicles

    Shuo Cheng, Bo Yang, Zheng Wang, and Kimihiko Nakano. Spatio-temporal image representation and deep- learning-based decision framework for automated vehicles. IEEE Transactions on Intelligent Transportation Systems , 23(12):24866–24875, 2022. 1

    work page 2022

  7. [7]

    Inductive bias of deep convolutional networks through pooling geometry

    Nadav Cohen and Amnon Shashua. Inductive bias of deep convolutional networks through pooling geometry. arXiv preprint arXiv:1605.06743, 2016. 2

    arXiv 2016

  8. [8]

    Pedestrian detection: A benchmark

    Piotr Doll ´ar, Christian Wojek, Bernt Schiele, and Pietro Per- ona. Pedestrian detection: A benchmark. In 2009 IEEE con- ference on computer vision and pattern recognition , pages 304–311. IEEE, 2009. 5

    work page 2009

Show all 59 references
  1. [9]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  2. [10]

    facebookresearch. fvcore. https://github.com/ facebookresearch/fvcore, 2019. 5

    2019

  3. [11]

    Simvp: Simpler yet better video prediction

    Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 3170–3180, 2022. 2, 3, 4, 6, 8

    2022

  4. [12]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recog- nition (CVPR), 2012. 5

    2012

  5. [13]

    Deep Learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, 2016. http : / / www . deeplearningbook.org. 3

    2016

  6. [14]

    Disentangling physi- cal dynamics from unknown factors for unsupervised video prediction

    Vincent Le Guen and Nicolas Thome. Disentangling physi- cal dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11474– 11484, 2020. 6, 1

    2020

  7. [15]

    Classifying pedestrian actions in advance using predicted video of urban driving scenes

    Pratik Gujjar and Richard Vaughan. Classifying pedestrian actions in advance using predicted video of urban driving scenes. In 2019 International Conference on Robotics and Automation (ICRA), pages 2097–2103. IEEE, 2019. 1

    2019

  8. [16]

    Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026–1034, 2015. 2

    2015

  9. [17]

    A dynamic multi-scale voxel flow network for video prediction

    Xiaotao Hu, Zhewei Huang, Ailin Huang, Jun Xu, and Shuchang Zhou. A dynamic multi-scale voxel flow network for video prediction. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6121–6131, 2023. 2, 3

    2023

  10. [18]

    Learning robot activities from first-person human videos using convolutional future regression

    Jangwon Lee and Michael S Ryoo. Learning robot activities from first-person human videos using convolutional future regression. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition Workshops, pages 1–2,

  11. [19]

    More convnets in the 2020s: Scal- ing up kernels beyond 51x51 using sparsity

    Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pechenizkiy, Decebal Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scal- ing up kernels beyond 51x51 using sparsity. arXiv preprint arXiv:2207.03620, 2022. 2, 3

    2022 arXiv

  12. [20]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 11976–11986,

  13. [21]

    Video frame synthesis using deep voxel flow

    Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In Proceedings of the IEEE international conference on computer vision, pages 4463–4471, 2017. 2, 3

    2017

  14. [22]

    Deep pre- dictive coding networks for video prediction and unsuper- vised learning

    William Lotter, Gabriel Kreiman, and David Cox. Deep pre- dictive coding networks for video prediction and unsuper- vised learning. arXiv preprint arXiv:1605.08104, 2016. 5, 6

    2016 arXiv

  15. [23]

    Predicting fu- ture occupancy grids in dynamic environment with spatio- temporal learning

    Khushdeep S Mann, Abhishek Tomy, Anshul Paigwar, Alessandro Renzaglia, and Christian Laugier. Predicting fu- ture occupancy grids in dynamic environment with spatio- temporal learning. In 2022 IEEE Intelligent Vehicles Sympo- sium (IV), pages 1121–1126. IEEE, 2022. 1

    2022

  16. [24]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Pa...

    2015

  17. [25]

    Rousseeuw

    Peter J. Rousseeuw. Silhouettes: A graphical aid to the inter- pretation and validation of cluster analysis. Journal of Com- putational and Applied Mathematics, 20:53–65, 1987. 7

    1987

  18. [26]

    Recog- nizing human actions: a local svm approach

    Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recog- nizing human actions: a local svm approach. In Proceedings of the 17th International Conference on Pattern Recognition,

  19. [27]

    Implicit stacked autoregressive model for video prediction

    Minseok Seo, Hakjin Lee, Doyi Kim, and Junghoon Seo. Implicit stacked autoregressive model for video prediction. arXiv preprint arXiv:2303.07849, 2023. 2, 1

    2023 arXiv

  20. [28]

    Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network

    Wenzhe Shi, Jose Caballero, Ferenc Husz ´ar, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on compu...

    2016

  21. [29]

    Convolutional lstm network: A machine learning approach for precipitation nowcasting

    Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. Advances in neural information processing sys- tems, 28, 2015. 2, 6, 1

    2015

  22. [30]

    Unsupervised learning of video representations using lstms

    Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudi- nov. Unsupervised learning of video representations using lstms. In International conference on machine learning , pages 843–852. PMLR, 2015. 4, 5

    2015

  23. [31]

    Sequence to sequence learning with neural networks

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014. 2

    2014

  24. [32]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1–9, 2015. 3

    2015

  25. [33]

    Temporal attention unit: To- wards efficient spatiotemporal predictive learning

    Cheng Tan, Zhangyang Gao, Lirong Wu, Yongjie Xu, Jun Xia, Siyuan Li, and Stan Z Li. Temporal attention unit: To- wards efficient spatiotemporal predictive learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18770–18782, 2023....

    2023

  26. [34]

    Open- stl: A comprehensive benchmark of spatio-temporal predic- tive learning

    Cheng Tan, Siyuan Li, Zhangyang Gao, Wenfei Guan, Ze- dong Wang, Zicheng Liu, Lirong Wu, and Stan Z Li. Open- stl: A comprehensive benchmark of spatio-temporal predic- tive learning. arXiv preprint arXiv:2306.11249, 2023. 2, 3, 4, 5, 6

    2023 arXiv

  27. [35]

    Mlp-mixer: An all-mlp architecture for vision

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lu- cas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:2...

  28. [36]

    Resmlp: Feedforward networks for image classification with data-efficient training

    Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izac- ard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions on ...

    2022

  29. [37]

    Patches are all you need? arXiv preprint arXiv:2201.09792, 2022

    Asher Trockman and J Zico Kolter. Patches are all you need? arXiv preprint arXiv:2201.09792, 2022. 3, 8

    2022 arXiv

  30. [38]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 3

    2017

  31. [39]

    Decomposing motion and con- tent for natural video sequence prediction

    Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and con- tent for natural video sequence prediction. arXiv preprint arXiv:1706.08033, 2017. 5

    2017 arXiv

  32. [40]

    Mcvd-masked conditional video diffusion for prediction, generation, and interpolation

    Vikram V oleti, Alexia Jolicoeur-Martineau, and Chris Pal. Mcvd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in Neural Informa- tion Processing Systems, 35:23371–23385, 2022. 3

    2022

  33. [41]

    Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning

    Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and S Yu Philip. Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In International Conference on Machine Learning, pages 5123–5132. PMLR, 2018. 2, 5, 6, 1

    2018

  34. [42]

    Eidetic 3d lstm: A model for video prediction and beyond

    Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Ming- sheng Long, and Li Fei-Fei. Eidetic 3d lstm: A model for video prediction and beyond. In International conference on learning representations, 2018. 5, 6, 1

    2018

  35. [43]

    Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms

    Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms. Advances in neural information processing systems, 30, 2017. 2, 5, 6, 1

    2017

  36. [44]

    Predrnn: A recurrent neural network for spatiotemporal predictive learn- ing

    Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jian- min Wang, S Yu Philip, and Mingsheng Long. Predrnn: A recurrent neural network for spatiotemporal predictive learn- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2208–2225, 2022. 6, 1

    2022

  37. [45]

    Memory in memory: A predictive neural network for learning higher-order non- stationarity from spatiotemporal dynamics

    Yunbo Wang, Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, and Philip S Yu. Memory in memory: A predictive neural network for learning higher-order non- stationarity from spatiotemporal dynamics. In Proceedings of the IEEE/CVF conference on computer vision and patter...

    2019

  38. [46]

    Sample-efficient reinforcement learning via conservative model-based actor-critic

    Zhihai Wang, Jie Wang, Qi Zhou, Bin Li, and Houqiang Li. Sample-efficient reinforcement learning via conservative model-based actor-critic. In Proceedings of the AAAI Con- ference on Artificial Intelligence , volume 36, pages 8612– 8620, 2022. 6

    2022

  39. [47]

    Theoretical analysis of the induc- tive biases in deep convolutional networks.Advances in Neu- ral Information Processing Systems, 36, 2024

    Zihao Wang and Lei Wu. Theoretical analysis of the induc- tive biases in deep convolutional networks.Advances in Neu- ral Information Processing Systems, 36, 2024. 2

    2024

  40. [48]

    Sample efficient deep reinforcement learning for dialogue systems with large action spaces

    Gell ´ert Weisz, Paweł Budzianowski, Pei-Hao Su, and Mil- ica Ga ˇsi´c. Sample efficient deep reinforcement learning for dialogue systems with large action spaces. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 26(11):2083–2097, 2018. 6

    2018

  41. [49]

    Pastnet: Introducing physical inductive biases for spatio-temporal video prediction

    Hao Wu, Wei Xion, Fan Xu, Xiao Luo, Chong Chen, Xian- Sheng Hua, and Haixin Wang. Pastnet: Introducing physical inductive biases for spatio-temporal video prediction. arXiv preprint arXiv:2305.11421, 2023. 2, 3

    2023 arXiv

  42. [50]

    Predcnn: Predictive learning with cascade convo- lutions

    Ziru Xu, Yunbo Wang, Mingsheng Long, Jianmin Wang, and M KLiss. Predcnn: Predictive learning with cascade convo- lutions. In IJCAI, pages 2940–2947, 2018. 2

    2018

  43. [51]

    Vptr: Efficient transformers for video prediction

    Xi Ye and Guillaume-Alexandre Bilodeau. Vptr: Efficient transformers for video prediction. In2022 26th International Conference on Pattern Recognition (ICPR) , pages 3492–

  44. [52]

    Crevnet: Conditionally reversible video prediction, 2019

    Wei Yu, Yichao Lu, Steve Easterbrook, and Sanja Fidler. Crevnet: Conditionally reversible video prediction, 2019. 5, 1

    2019

  45. [53]

    Efficient and information-preserving future frame prediction and beyond

    Wei Yu, Yichao Lu, Steve Easterbrook, and Sanja Fidler. Efficient and information-preserving future frame prediction and beyond. 2020. 6

    2020

  46. [54]

    Predicting citywide crowd flows using deep spatio-temporal residual networks

    Junbo Zhang, Yu Zheng, Dekang Qi, Ruiyuan Li, Xiuwen Yi, and Tianrui Li. Predicting citywide crowd flows using deep spatio-temporal residual networks. Artificial Intelli- gence, 259:147–166, 2018. 4, 5

    2018

  47. [55]

    Pose-forecasting aided human video prediction with graph convolutional networks

    Yang Zhao and Yong Dou. Pose-forecasting aided human video prediction with graph convolutional networks. IEEE Access, 8:147256–147264, 2020. 1 The supplementary material provides a comprehensive analysis of the STLight method. Each section contributes unique insights: In Secti...

    2020

  48. [57]

    • Repeated STLMixer The parameter count from our proposed STLMixer architecture is O de · (d2 + d · k2 T1 + d · k2 T2 )

    Therefore STLight decodes the processed signals with a parameter count of O d/p2 · (T ′C) · k2 D = O d/p2 · (T ′C) . • Repeated STLMixer The parameter count from our proposed STLMixer architecture is O de · (d2 + d · k2 T1 + d · k2 T2 ) . In fact, each STLMixer is composed of ...

  49. [58]

    Similarly k2 T1 ≪ d and k2 T2 ≪ d

    T · C and T ′ · C typically remain below 10, while d often exceeds 1000, hence T · C ≪ d and T ′ · C ≪ d. Similarly k2 T1 ≪ d and k2 T2 ≪ d

  50. [59]

    dis- persion

    O ≤ 2 and p ≤ 2, leading to kE = p · max(1, O) ≤ 4 using the formula shown in Section 2.1. Therefore the parameter counts of our encoder and decoder blocks scale linearly with respect to d, while the parameters count of the repeated STLMixer blocks can be expressed as O(de · d...

    2000

  51. [2004]

    IEEE, 2004

    ICPR 2004., volume 3, pages 32–36. IEEE, 2004. 4, 5

    2004

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.