Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Unstructured magnitude pruning can remove about half of the weights from most Transformer-based time-series forecasters without meaningful loss of accuracy, while structured pruning fails to deliver real inference speedup.

desk verdict A useful but uneven benchmark: the 50% sparsity finding holds, but the no-speedup claim overstates what was actually measured. read the letter →

arxiv 2412.12883 v1 pith:CCEFPV5D submitted 2024-12-17 cs.LG cs.AI

Nicholas Kiefer , Arvid Weyrauch , Muhammed Öz , Achim Streit , Markus Götz , Charlotte Debus This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0762M10
keywords networkpruningTransformertimeseriesforecastingunstructuredstructuredDepGraphmodelcompressioninferencespeedup
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether pruning can make Transformer-based multivariate time-series forecasters cheaper to deploy without wrecking their accuracy. It reports that unstructured magnitude pruning, which zeroes out the smallest weights, removes roughly 50% of parameters from most of the five tested models with no significant loss in test MSE, and that the frequency-domain models Autoformer and FEDformer stay usable even at 90% sparsity. It also reports that fine-tuning the pruned models is needed to recover accuracy, and that structured node-level pruning with the DepGraph method, despite cutting FLOP counts, produces little or no inference speedup on A100/H100 GPUs. The practical message is that model-size reduction alone does not translate into faster inference for these architectures, so deployment gains must come from sparsity-aware runtimes or other compression schemes.

What carries the argument

The mechanism that carries the argument is a train-prune-fine-tune loop applied uniformly across architectures. Unstructured pruning uses a binary mask computed by PyTorch, zeroing the smallest-magnitude weights at ten sparsity levels spread in log-space ($s = 1 - 0.8^i$), with the mask retained during fine-tuning. Structured pruning uses the dependency-graph pruner from torch-pruning (DepGraph), which groups parameters by their dependencies in the forward pass and removes whole nodes (rows or columns), with density measured after pruning rather than assumed. The evaluation machinery is the comparison of test MSE, parameter density, FLOP reduction, and CUDA-event-timed inference speedup at batch size 32, plus two control studies (scaling model size down on ETTm2 and dataset size up on ENTSO-E) meant to disentangle overfitting from pruning effects.

What would settle it

Re-implement the five models with standard attention so that TensorRT or a similar sparse-kernel compiler can compile them, then measure structurally pruned inference on a low-power embedded device at batch sizes 1 and 32; if pruned Autoformer or FEDformer shows a wall-clock speedup comparable to its FLOP reduction (say above 2x), the paper's no-significant-time-savings claim for structured pruning would be falsified in that setting.

Watch

Extended reading notes

Core claim

The central discovery is a benchmark result: magnitude-based unstructured pruning is a broadly safe operation for Transformer time-series forecasters, with a 50% sparsity threshold holding across Transformer, Informer, Autoformer, FEDformer, and Crossformer on ETT, ECL, Exchange, Traffic, Weather, and ENTSO-E datasets, while the Fourier-decomposition models Autoformer and FEDformer tolerate pruning to high sparsity before losing competitive accuracy. A corollary is that the pruned models sometimes beat their dense counterparts, which the authors attribute to a regularization effect rather than a lottery-ticket phenomenon. The second discovery is negative: structured pruning via DepGraph does remove parameters and reduce FLOPs by up to a factor of about seven, yet measured inference speedup tops out around 1.5x for Informer and is near 1.0 or below for others, because the time-series-specific attention and decomposition modules dominate runtime and cannot exploit the shrunken weight matrices.

Load-bearing premise

The conclusion that structured pruning offers no significant time savings assumes that GPU timings with PyTorch's dense CUDA kernels, at batch size 32, capture the deployment setting that motivates pruning; because TensorRT compilation failed for every model, no optimized sparse kernel was ever measured.

Editorial extensions

If this is right

  • With the 50% sparsity threshold confirmed across five architectures and multiple datasets, practitioners can halve the parameter count of time-series Transformers at negligible accuracy cost, and Fourier-style models can be cut to 10% of parameters when memory is the binding constraint.
  • Because fine-tuning is required for most models, pruning should be part of a train-prune-fine-tune pipeline rather than an after-the-fact inference trick.
  • Structured pruning's FLOP reductions (up to 7x for Informer) do not deliver commensurate wall-clock speedups on dense CUDA kernels, so FLOP counts alone overstate deployment benefits.
  • On small datasets, smaller models match or beat large pruned ones, so the efficient choice is to train a small model directly; on large datasets, training big then pruning retains accuracy better than training small from scratch.
  • The negative structured-pruning result localizes the runtime bottleneck: time-series-specific components such as decomposition, frequency transforms, and cross-dimension attention dominate inference, not the prunable linear layers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the 'no significant time savings' conclusion is tied to the authors' measurement setup (A100/H100 GPUs, PyTorch native CUDA kernels, batch size 32, and a failed TensorRT compilation); on embedded CPUs or with sparse-inference engines the speedup picture could differ, and the paper itself flags this as future work.
  • Editorial extension: the finding that pruned models can outperform dense ones on small datasets suggests a regularization story; a direct test would be to compare pruning at 50% sparsity against standard weight decay or dropout with matched compute.
  • Editorial extension: the DepGraph failure to reach target sparsity on Autoformer and FEDformer indicates that dependency-graph grouping interacts badly with decomposition modules; a structured pruner that operates on attention heads or frequency bands might fare better.
  • Editorial extension: the benchmark suggests lottery-ticket search, finding sparse subnetworks before training, may be worthwhile for these architectures, since magnitude pruning finds usable subnetworks after training; the authors mention dynamic sparse training as an open direction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents a benchmark study of unstructured magnitude pruning and structured DepGraph pruning applied to five Transformer-based multivariate time-series forecasting models (Transformer, Informer, Autoformer, FEDformer, Crossformer) across several public datasets and forecasting horizons. The authors evaluate predictive performance (MSE), parameter density, FLOPs, and inference-time speedup, and add experiments with reduced model size, a larger ENTSO-E dataset, and fine-tuning after pruning. The main reported findings are that most models tolerate about 50% unstructured sparsity without major loss, Fourier-based models remain usable up to about 90% sparsity, fine-tuning is needed to recover performance, and structured pruning does not yield meaningful inference speedups even with claimed hardware and software support.

Significance. If the benchmark's conclusions are supported, this would be a useful reference for practitioners choosing pruning strategies for Transformer-based time-series models. The study's breadth is a strength: five architectures, multiple public datasets and horizons, plus dedicated ablations on model size (ETTm2) and dataset size (ENTSO-E). The authors also report failures openly, including TensorRT compilation issues for all models, which is informative for reproducibility. However, the headline speedup claim is not supported by the reported measurements, and the lack of uncertainty quantification weakens the quantitative conclusions. The benchmark's predictive-performance findings are plausible and potentially valuable, but the central claims need to be either strengthened or substantially qualified.

major comments (4)
  1. [Abstract and Section 5] The abstract states that 'even with corresponding hardware and software support, structured pruning is unable to provide significant time savings,' but Section 5 reports that TensorRT compilation failed for all models and that the authors 'refrained' from custom implementations. No optimized sparse kernel or compiled deployment path was ever measured. The speedups in Table 3 were obtained with native PyTorch CUDA kernels acting on the reduced dense matrices, which is not 'hardware and software support' in the sense claimed. This is an internal contradiction between the headline claim and the experimental scope. Please either measure an actual sparse/compiled inference path or restrict the claim to the kernels and runtime environment actually used.
  2. [Section 3.1 and Table 3] Although Section 3.1 states that all models are trained three times, Figures 1 and 2 show only point estimates with no error bars or confidence intervals, and Table 3 reports a single speedup value per model from one timing run (500 batches after 50 warm-up steps). Consequently, differences among speedups such as Transformer 1.21 vs. Crossformer 1.22, and the conclusion that only Informer shows a meaningful speedup, are not statistically supported. Please provide repeated timing measurements with standard deviations and include error bars or uncertainty bands in the pruning curves.
  3. [Section 4.4 and Table 4] The fine-tuning experiments are limited to a single dataset (ETTm1), a single forecasting horizon (192), and a single target density (33%). The results are also mixed: the Transformer already improves with pruning alone (0.6869 vs. 0.7256) and fine-tuning degrades it to 0.7749, while Crossformer recovers from 0.8124 to 0.3502. This single configuration does not support the abstract's general claim that 'fine-tuning pruned models is necessary.' Either extend the fine-tuning study to additional datasets/horizons or qualify the claim to the configuration tested.
  4. [Section 5 and Figures 1-3] The conclusion that Fourier-based models (Autoformer, FEDformer) 'retain sufficient predictive performance up to 90% sparsity' is not backed by an explicit performance criterion. The paper never defines what loss increase is considered acceptable, so 'sufficient' is not falsifiable. I recommend specifying a quantitative criterion (e.g., relative MSE increase below a stated threshold compared to the dense baseline) and applying it consistently when summarizing the sparsity robustness results.
minor comments (5)
  1. [Section 3.3, Eq. (2)] Equation (2) defines s = 1 - 0.8^i for i = 0,...,10, which gives eleven values including the unpruned case, not 'ten target sparsity levels.' Please clarify the count and the exact levels used.
  2. [Section 4.3] The phrase 'pruned to a density of d = 0.85 ≈ 0.33' is confusing: Eq. (2) defines sparsity, not density, and 0.85 does not correspond to a density of 0.33. State the target sparsity and density explicitly and reconcile the notation with Eq. (2).
  3. [Section 3.6] The text says the reduced-model experiments use 'ETTm2 (corresponding to our highest pruning rate),' but the preceding pruning experiments are mainly reported on ETTm1. Clarify why ETTm2 was chosen for this ablation.
  4. [Figures 1 and 2] The figures are dense and the caption says 'Best viewed zoomed in.' Consider plotting each model in a separate panel or using more distinguishable line styles so the results are readable in print.
  5. [Table 5] The column header 'ENTSO-E (large)' is misleading because the table lists both small and large variants for that dataset. Consider renaming the columns to reflect the two datasets and the two model sizes more clearly.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the benchmark compares trained baselines against pruned models on held-out test data; the only author-overlap citations are non-essential related-work references.

full rationale

The paper is an empirical benchmark, not a derivation. Unstructured magnitude pruning is applied to trained models at pre-defined sparsity levels s = 1 - 0.8^i, and performance is measured by MSE on held-out test data; no parameter is fitted to the test set and then renamed a prediction. The structured-pruning speedup numbers (Table 3) are direct CUDA-event time ratios, not quantities implied by construction from the pruning ratio. The claimed 'even with corresponding hardware and software support' conclusion is undermined by the paper's own Section 5 admission that 'the published implementations failed to compile for all models' and that the authors 'refrained from this step'; however, this is an evidentiary and generalization gap, not circularity, since the claim is not made true by definition. The only self-citations with author overlap ([33] on pruning-after-training, [44] on an application example) appear in related-work context and are not load-bearing; the pruning-after-training claim is independently supported by [12] and by the paper's own fine-tuning experiments. The central predictive-performance findings are self-contained against external benchmarks and openly report failures (e.g., FEDformer training instability, DepGraph failing to reach target sparsity). Consistent with the rubric that self-citation is not circularity unless it carries the argument, no circular step is exhibited, so a low score is appropriate despite the validity concern about the speedup generalization.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The benchmark has no fitted parameters in the mathematical sense. The hand-chosen experimental settings that shape the conclusions are the sparsity schedule, the model scale-down factor, and the ENTSO-E reduced model size. The key domain assumptions are that MSE captures performance, that global density fractions are comparable across models, that DepGraph correctly groups parameters in time-series Transformers, and that GPU latency measurements represent the deployment hardware the paper motivates.

free parameters (3)
  • sparsity schedule base = s = 1 - 0.8^i for i = 0..10
    Ten target sparsity levels spread uniformly in log-space, chosen by hand in Section 3.3. The results are reported against these levels.
  • model scale-down factor for small variants = 10x smaller linear embedding layer
    Section 3.6 scales models down by a factor of 10 on ETTm2 to study overfitting. The factor is arbitrary and affects the small-model comparison.
  • ENTSO-E reduced model size = 50% of original parameter count
    Section 4.6 chooses the reduced model size as the largest pruning ratio without performance loss, based on Figure 3. This choice follows from the pruning results rather than an external rule.
assumptions (5)
  • domain assumption MSE test loss is the appropriate and sufficient measure of predictive performance
    Section 3.5 defines evaluation only by MSE, with no interval or calibration metrics and no displayed variance despite three training runs.
  • domain assumption Global pruning ratios as fractions of total parameters allow fair comparison across models
    Section 3.3 notes a fixed pruning ratio does not imply equal parameter budget; models with different sizes are compared at the same density fraction.
  • domain assumption DepGraph dependency graph correctly captures all prunable parameter groups in time-series Transformers
    Structured pruning relies on torch-pruning's DepGraph in Section 3.3; if the graph misses time-series-specific modules, the structured pruning results may be artifacts of the pruner.
  • domain assumption Inference timing on A100/H100 GPUs with PyTorch CUDA kernels represents target deployment hardware
    The motivation is low-power embedded devices in Section 1, but Table 3 measurements are on data-center GPUs with native PyTorch kernels.
  • domain assumption Training settings adopted from FEDformer and Crossformer publications are appropriate for all five models
    Section 3.1 applies one hyperparameter configuration to the vanilla Transformer, Informer, Autoformer, FEDformer, and Crossformer, which may disadvantage models whose published recipes differ.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting." pith.science (2026). https://pith.science/paper/CCEFPV5D

@misc{pith2026241212883,
  author       = {Pith},
  title        = {Pith review of: A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CCEFPV5D}},
  note         = {Machine review of arXiv:2412.12883}
}
read the original abstract

The current landscape in time-series forecasting is dominated by Transformer-based models. Their high parameter count and corresponding demand in computational resources pose a challenge to real-world deployment, especially for commercial and scientific applications with low-power embedded devices. Pruning is an established approach to reduce neural network parameter count and save compute. However, the implications and benefits of pruning Transformer-based models for time series forecasting are largely unknown. To close this gap, we provide a comparative benchmark study by evaluating unstructured and structured pruning on various state-of-the-art multivariate time series models. We study the effects of these pruning strategies on model predictive performance and computational aspects like model size, operations, and inference time. Our results show that certain models can be pruned even up to high sparsity levels, outperforming their dense counterpart. However, fine-tuning pruned models is necessary. Furthermore, we demonstrate that even with corresponding hardware and software support, structured pruning is unable to provide significant time savings.

Figures

Figures reproduced from arXiv: 2412.12883 by the authors.

Figure 1
Figure 1. Pruning results for weight magnitude pruning. Plotted is the MSE on the test dataset over the parameter [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. We observe, that while the vanilla Transformer and the Informer can be pruned to 1% of [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 2
Figure 2. Pruning results for structured node pruning using [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Pruning results for weight magnitude pruning on the ENTSO-E test dataset with prediction length [PITH_FULL_IMAGE:figures/full_fig_p012_3.png]
Figure 4
Figure 4. Figure 4: Loss curves for all models during training on the ENTSO-E dataset, averaged over 50 steps to reduce [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs

    cs.LG 2025-09 conditional novelty 5.0 of 10

    A training-free method that prunes low-magnitude dimensions of projected query/key vectors in attention, cutting dot-product cost by 25% with small benchmark degradation.

Reference graph

Works this paper leans on

51 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    Maximiliana Behnke and Kenneth Heafield. 2020. Losing Heads in the Lottery: Pruning Transformer Attention in Neural Machine Translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 2664–2674. ...

    work page doi:10.18653/v1/2020.emnlp-main.211 2020

  2. [2]

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2023. Token Merging: Your ViT But Faster. https://doi.org/10.48550/arXiv.2210.09461 arXiv:2210.09461 [cs]

    arXiv doi:10.48550/arxiv.2210.09461 2023

  3. [3]

    Feiyang Chen, Ziqian Luo, Lisang Zhou, Xueting Pan, and Ying Jiang. 2024. Comprehensive Survey of Model Compression and Speed up for Vision Transformers. https://doi.org/10.48550/arXiv.2404.10407 arXiv:2404.10407 [cs]

    arXiv doi:10.48550/arxiv.2404.10407 2024

  4. [4]

    Shengchao Chen, Guodong Long, Jing Jiang, and Chengqi Zhang. 2024. Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models. arXiv:2405.20348 [physics.ao-ph] https://arxiv.org/abs/2405. 20348

    arXiv 2024

  5. [5]

    Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. 2023. A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations. https://doi.org/10.48550/arXiv.2308.06767 arXiv:2308.06767 [cs]

    arXiv doi:10.48550/arxiv.2308.06767 2023

  6. [6]

    Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. 2023. A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations. arXiv:2308.06767 [cs.LG] https://arxiv.org/abs/2308.06767

    arXiv 2023

  7. [7]

    Krishna Teja Chitty-Venkata, Sparsh Mittal, Murali Emani, Venkatram Vishwanath, and Arun K. Somani. 2023. A survey of techniques for optimizing transformer inference. Journal of Systems Architecture 144 (Nov. 2023), 102990. https://doi.org/10.1016/j.sysarc.2023.102990

    arXiv 2023

  8. [8]

    Jan Debus, Charlotte Debus, Günther Dissertori, and Markus Götz. 2024. PETNet–Coincident Particle Event Detection using Spiking Neural Networks. In 2024 Neuro Inspired Computational Elements Conference (NICE) . IEEE, La Jolla, CA, USA, 1–9. https://doi.org/10.1109/NICE61972.2024.10549584

    arXiv 2024

Show all 51 references
  1. [9]

    Dazhao Du, Bing Su, and Zhewei Wei. 2022. Preformer: Predictive Transformer with Multi-Scale Segment-wise Correlations for Long-Term Time Series Forecasting. arXiv:2202.11356 [cs.LG] https://arxiv.org/abs/2202.11356

    2022 arXiv

  2. [10]

    Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. 2023. DepGraph: Towards Any Structural Pruning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE Computer Society, Vancouver, BC, Canada, 16091–16101. https://doi.org/10....

    2023

  3. [11]

    Quentin Fournier, Gaétan Marceau Caron, and Daniel Aloise. 2023. A Practical Survey on Faster and Lighter Trans- formers. Comput. Surveys 55, 14s (July 2023), 304:1–304:40. https://doi.org/10.1145/3586074

    2023 doi

  4. [12]

    Jonathan Frankle and Michael Carbin. 2019. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. https://doi.org/10.48550/arXiv.1803.03635 arXiv:1803.03635 [cs]

    2019 doi arXiv

  5. [13]

    Jianhua Gao, Bingjie Liu, Weixing Ji, and Hua Huang. 2024. A Systematic Literature Survey of Sparse Matrix-Vector Multiplication. arXiv preprint arXiv:2404.06047 55, 244 (2024), 36 pages

    2024 arXiv

  6. [14]

    Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 2018. 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks. CVPR 1 (2018), 11 pages

    2018

  7. [15]

    Hagiwara

    M. Hagiwara. 1993. Removal of hidden units and weights for back propagation networks. In Proceedings of 1993 inter- national conference on neural networks (IJCNN-93-nagoya, japan) , Vol. 1. Proceedings of 1993 International Conference on Neural, Nagoya, Japan, 351–354 vol.1. h...

    1993

  8. [16]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs.CV] https://arxiv.org/abs/1512.03385

    2015 arXiv

  9. [17]

    Yang He and Lingao Xiao. 2024. Structured Pruning for Deep Convolutional Neural Networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 2024.05 (2024), 1–20. https://doi.org/10.1109/TPAMI. 2023.3334614 arXiv:2303.00566 [cs]

    2024

  10. [18]

    Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks. https://doi.org/10.48550/arXiv.2102.00554 , Vol. 1, No. 1, Article . Publication date: D...

    2021 doi arXiv

  11. [19]

    Ruibing Jin, Qing Xu, Min Wu, Yuecong Xu, Dan Li, Xiaoli Li, and Zhenghua Chen. 2024. LLM-based Knowledge Pruning for Time Series Data Analytics on Edge-computing Devices. arXiv:2406.08765 [cs.LG] https://arxiv.org/abs/2406.08765

    2024 arXiv

  12. [20]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv:2001.08361 [cs.LG] https://arxiv.org/abs/2001.08361

    2020 arXiv

  13. [21]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. https://doi.org/10.48550/arXiv.2001. 08361 arXiv:2001.08361 [cs, stat]

    2020 doi arXiv

  14. [22]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs.LG] https://arxiv.org/abs/1412.6980

    2017 arXiv

  15. [23]

    Yann LeCun, John Denker, and Sara Solla. 1989. Optimal Brain Damage. In Advances in Neural Information Processing Systems, D. Touretzky (Ed.), Vol. 2. Morgan-Kaufmann, San Francisco, CA, USA. https://proceedings.neurips.cc/paper_ files/paper/1989/file/6c9882bbac1c7093bd2504188...

    1989

  16. [24]

    Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. 2021. Pruning and quantization for deep neural network acceleration: A survey.Neurocomputing 461 (Oct. 2021), 370–403. https://doi.org/10.1016/j.neucom.2021.07.045

    2021 doi

  17. [25]

    Bohan Liu, Zijie Zhang, Peixiong He, Zhensen Wang, Yang Xiao, Ruimeng Ye, Yang Zhou, Wei-Shinn Ku, and Bo Hui

  18. [26]

    Liu, and Schahram Dustdar

    Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dustdar. 2022. Pyraformer: Low- Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting. In International Conference on Learning Representations. ICLR, virtual, 20 pag...

    2022

  19. [27]

    Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. 2024. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations. ICLR, Vienna, Austria, 1–25. https:...

    2024

  20. [28]

    Jiachen Mao, Huanrui Yang, Ang Li, Hai Li, and Yiran Chen. 2021. TPrune: Efficient Transformer Pruning for Mobile Devices. ACM Trans. Cyber-Phys. Syst. 5, 3, Article 26 (apr 2021), 22 pages. https://doi.org/10.1145/3446640

    2021 doi

  21. [29]

    Tung Nguyen, Rohan Shah, Hritik Bansal, Troy Arcomano, Sandeep Madireddy, Romit Maulik, Veerabhadra Kotamarthi, Ian Foster, and Aditya Grover. 2023. Scaling transformer neural networks for skillful and reliable medium-range weather forecasting. arXiv:2312.03876 [physics.ao-ph]...

    2023 arXiv

  22. [30]

    Nowak, Bram Grooten, Decebal Constantin Mocanu, and Jacek Tabor

    Aleksandra I. Nowak, Bram Grooten, Decebal Constantin Mocanu, and Jacek Tabor. 2023. Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training. https://doi.org/10.48550/arXiv.2306.12230 arXiv:2306.12230 [cs, stat]

    2023 doi arXiv

  23. [31]

    NVIDIA. 2024. TensorRT. https://github.com/NVIDIA/TensorRT. Accessed: 2024-11-23

    2024

  24. [32]

    European Network of Transmission System Operators for Electricity. 2024. ENTSO-E. Transparency Platform. https: //transparency.entsoe.eu. Accessed: 2024-11-23

    2024

  25. [33]

    Muhammed Öz, Nicholas Kiefer, Charlotte Debus, Jasmin Hörter, Achim Streit, and Markus Götz. 2024. Model Fusion via Neuron Transplantation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases . Springer, Springer, virtual, 3–19

    2024

  26. [34]

    Lorenzo Papa, Paolo Russo, Irene Amerini, and Luping Zhou. 2024. A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking.IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 1 (2024), 1–20. https://doi.org/10.1109/TPAMI.202...

    2024

  27. [35]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  28. [36]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fan...

    2019 arXiv

  29. [37]

    Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, and Dacheng Tao. 2024. A Survey on Transformer Compression. https://doi.org/10.48550/arXiv.2402.05964 arXiv:2402.05964 [cs]

    2024 doi arXiv

  30. [38]

    J. F. Torres, A. M. Fernández, A. Troncoso, and F. Martínez-Álvarez. 2017. Deep Learning-Based Approach for Time Series Forecasting with Application to Electricity Load. In Biomedical Applications Based on Natural and Artificial Computing, José Manuel Ferrández Vicente, José R...

    2017

  31. [39]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. https://doi.org/10.48550/arXiv.1706.03762 arXiv:1706.03762 [cs] version: 7

    2023 doi arXiv

  32. [40]

    Huan Wang, Can Qin, Yue Bai, Yulun Zhang, and Yun Fu. 2022. Recent Advances on Neural Network Pruning at Initialization. https://doi.org/10.48550/arXiv.2103.06460 arXiv:2103.06460 [cs]

    2022 doi arXiv

  33. [41]

    Yuqing Wang, Yun Zhao, and Linda Petzold. 2022. Enhancing Transformer Efficiency for Multivariate Time Series Classification. https://doi.org/10.48550/arXiv.2203.14472 arXiv:2203.14472 [cs]

    2022 doi arXiv

  34. [42]

    Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. 2023. Transformers in Time Series: A Survey. https://doi.org/10.48550/arXiv.2202.07125 arXiv:2202.07125 [cs, eess, stat]

    2023 doi arXiv

  35. [43]

    Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. 2023. Transformers in Time Series: A Survey. arXiv:2202.07125 [cs.LG] https://arxiv.org/abs/2202.07125

    2023 arXiv

  36. [44]

    Weyrauch, T

    A. Weyrauch, T. Steens, O. Taubert, B. Hanke, A. Eqbal, E. Gotz, A. Streit, M. Gotz, and C. Debus. 2024. ReCycle: Fast and Efficient Long Time Series Forecasting with Residual Cyclic Transformers. In 2024 IEEE Conference on Artificial Intelligence (CAI). IEEE Computer Society,...

    2024

  37. [45]

    Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2022. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. https://doi.org/10.48550/arXiv.2106.13008 arXiv:2106.13008 [cs]

    2022 doi arXiv

  38. [46]

    Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023. Are Transformers Effective for Time Series Forecasting? Proceedings of the AAAI Conference on Artificial Intelligence 37, 9 (Jun. 2023), 11121–11128. https://doi.org/10.1609/aaai. v37i9.26317

    2023 doi

  39. [47]

    Yunhao Zhang and Junchi Yan. 2022. Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting. InThe Eleventh International Conference on Learning Representations. ICLR 2023, Kigali Rwanda,

    2022

  40. [48]

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. https://doi.org/10.48550/arXiv.2012.07436 arXiv:2012.07436 [cs]

    2021 doi arXiv

  41. [49]

    https://openreview.net/forum?id=vSVLM2j9eie

  42. [51]

    Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. FEDformer: Frequency Enhanced De- composed Transformer for Long-term Series Forecasting. https://doi.org/10.48550/arXiv.2201.12740 arXiv:2201.12740 [cs, stat]. , Vol. 1, No. 1, Article . Publication d...

    2022 doi arXiv

  43. [2024]

    arXiv:2403.04861 [cs.LG] https://arxiv.org/abs/2403.04861

    A Survey of Lottery Ticket Hypothesis. arXiv:2403.04861 [cs.LG] https://arxiv.org/abs/2403.04861

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.