REVIEW 12 cited by
Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Time series foundation models have demonstrated impressive performance as zero-shot forecasters. However, achieving effectively unified training on time series remains an open challenge. Existing approaches introduce some level of model specialization to account for the highly heterogeneous nature of time series data. For instance, Moirai pursues unified training by employing multiple input/output projection layers, each tailored to handle time series at a specific frequency. Similarly, TimesFM maintains a frequency embedding dictionary for this purpose. We identify two major drawbacks to this human-imposed frequency-level model specialization: (1) Frequency is not a reliable indicator of the underlying patterns in time series. For example, time series with different frequencies can display similar patterns, while those with the same frequency may exhibit varied patterns. (2) Non-stationarity is an inherent property of real-world time series, leading to varied distributions even within a short context window of a single time series. Frequency-level specialization is too coarse-grained to capture this level of diversity. To address these limitations, this paper introduces Moirai-MoE, using a single input/output projection layer while delegating the modeling of diverse time series patterns to the sparse mixture of experts (MoE) within Transformers. With these designs, Moirai-MoE reduces reliance on human-defined heuristics and enables automatic token-level specialization. Extensive experiments on 39 datasets demonstrate the superiority of Moirai-MoE over existing foundation models in both in-distribution and zero-shot scenarios. Furthermore, this study conducts comprehensive model analyses to explore the inner workings of time series MoE foundation models and provides valuable insights for future research.
Forward citations
Cited by 12 Pith papers
-
Byte Pair Encoding for Efficient Time Series Forecasting
A byte-pair-encoding tokenizer that converts repeated temporal motifs into single tokens improves zero-shot forecasting accuracy and speed over sample-wise and patch-based methods.
-
Learning Spatio-Temporal Foundation Models from Pure Synthetic Data
A spatio-temporal foundation model pre-trained exclusively on synthetic stochastic graph dynamics outperforms real-data-pretrained STFMs in zero-shot traffic forecasting, according to the paper's benchmarks.
-
Auditable Context-Aware HFMD Forecasting with Structured LLM Agents
An LLM-based two-agent system can forecast hand-foot-mouth disease cases with accuracy comparable to top numerical models while generating human-readable risk explanations.
-
DIVER-0 : A Fully Channel Equivariant EEG Foundation Model
A channel-permutation-equivariant EEG transformer with full spatio-temporal attention achieves competitive BCI performance with only 10% of pretraining data.
-
Time Series Representations for Classification Lie Hidden in Pretrained Vision Transformers
Frozen vision transformers, applied to image representations of time series, produce classification features that outperform or match time series foundation models on UCR and UEA benchmarks.
-
Mixture-of-Experts for Personalized and Semantic-Aware Next Location Prediction
NextLocMoE combines a location-semantics expert mixer and a persona-based expert mixer inside a trimmed LLaMA backbone to set new accuracy marks on next-location prediction benchmarks.
-
BLAST: Balanced Sampling Time Series Corpus for Universal Forecasting Models
A balanced sampling strategy over statistically characterized time series patterns lets universal forecasting models train on 78 billion tokens instead of 419 billion, with equal or better zero-shot accuracy.
-
Towards a Foundation Model for Communication Systems
A single pre-trained transformer can forecast and interpolate multiple wireless channel features (rank, precoder, Doppler, delay) on simulated 5G NR data.
-
Hopformer: Homogeneity-Pursuit Transformer for Time Series Forecasting
A two-stage forecaster (SPA trend extraction + LoRA-fine-tuned residual Transformer) that the paper claims beats prior models by 6.56% MASE, though the claim is not robust to its own extended baseline tables.
-
AdaPTS: Adapting Univariate Foundation Models to Probabilistic Multivariate Time Series Forecasting
Adapters that map multivariate time series into a latent space let a frozen univariate foundation model produce probabilistic multivariate forecasts, improving MSE on most tested tasks.
-
N-BEATS-MOE: N-BEATS with a Mixture-of-Experts Layer for Heterogeneous Time Series Forecasting
Adding a gating network on top of N-BEATS block outputs gives modest SMAPE improvements on some heterogeneous benchmark series, but the gains are small and not statistically validated.
-
Comparative Analysis of Time Series Foundation Models for Demographic Forecasting: Enhancing Predictive Accuracy in US Population Dynamics
TimesFM achieved the lowest MSE on 13 of 15 US state-race population forecast tasks, but the paper fine-tuned it for 50 epochs per state despite claiming no task-specific fine-tuning.
Discussion (0). Continue with ORCID to comment.