Pith. sign in

REVIEW 4 major objections 6 minor 55 references

Mixture of Low Rank Adaptation with Partial Parameter Sharing for Time Series Forecasting

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Standard multi-step forecasting reads every horizon from one shared representation; this paper proves that construction carries a strictly positive minimum error when the horizon exceeds the representation dimension, and introduces a…

desk verdict The MoLA empirical recipe is plausible and mostly works, but the expressiveness-bottleneck theorem is not established as stated. read the letter →

arxiv 2505.17872 v2 pith:YFTOHXM7 submitted 2025-05-23 cs.LG cs.AI

Licheng Pan , Zhichao Chen , Haoxuan Li , Guangyi Liu , Zhijian Xu , Zhaoran Liu , Hao Wang , Ying Wei This is my paper
classification cs.LGcs.AI
keywords time-seriesforecastingexpressivenessbottleneckmulti-tasklow-rankadaptationmixtureofexpertsparameter-efficientfine-tuninglong-horizon
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the dominant multi-task forecasting (MT-F) design—one encoder representation $R$ read out by a linear layer into all $T$ forecast steps—has an expressiveness bottleneck: when $T > L+1$, the minimum attainable error is strictly positive and independent of how good the encoder is. If true, this would mean every shared-representation multi-step decoder is irreducibly lossy, so gains must come from the decoding side. To bypass the bottleneck, the paper proposes pre-training a one-step foundation model and then adapting it with low-rank (LoRA) modules that are specific to each forecast horizon. The Mixture-of-LoRA (MoLA) variant segments the horizon and shares LoRA experts across segments with learned weights, preserving parameter sharing while removing the shared-representation constraint. On standard benchmarks, MoLA applied to iTransformer improves MSE/MAE over the base model and over state-of-the-art baselines on most settings.

What carries the argument

The load-bearing object is the 'expressiveness bottleneck': the projection of the label sequence $Y$ onto the null space of the decoder weight matrix $\bar{W}$, which cannot be reached by any decoder output and therefore contributes unavoidable squared error $\|U_t^\top Y\|_2^2$ per lost direction. The paper's remedy is a two-stage adaptation: first train a foundation model with a single-step output (where the null space vanishes), then freeze it and inject LoRA matrices $B^{(p)}A^{(p)}$ into selected layers for each forecast segment, with per-segment mixture weights $\Delta_k$ combining a shared pool of $P$ LoRA experts. This keeps the number of decoder outputs per segment at $S=T/K\leq L+1$, so the bottleneck term disappears, while the shared experts capture inter-step correlations.

What would settle it

Train a standard MT-F model on a single label sequence whose $T$ future values are chosen so each row lies in $\mathbb{R}^{L+1}$, jointly optimizing encoder and decoder weights; if the training residual can be driven to zero while $T > L+1$, the claimed representation-independent error floor does not appear in the actual training objective. A second check: with a multivariate embedding of dimension $L\times D > T$, verify whether the theorem's predicted positive minimum error is observable at all.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 3.1: with a fixed linear decoder $\bar{W}\in\mathbb{R}^{T\times(L+1)}$, the least-squares forecast error is $\|\epsilon\|_F^2 = \sum_{t=\mathrm{rank}(\bar{W})+1}^{T}\|U_t^\top Y\|_2^2$, where $\{U_t\}$ span the null space of $\bar{W}$; this error is positive whenever $T>L+1$ and is independent of the representation $R$ the encoder provides. The authors therefore argue that sharing one representation across all forecast steps caps achievable accuracy, and that the remedy is to give each step (or segment of steps) its own adapted representation while freezing the pretrained backbone. They report that MoLA outperforms both the MT-F and autoregressive paradigms on the tested datasets and that the benefit grows with horizon length.

Load-bearing premise

The proof of the unavoidable error fixes the decoder weights and optimizes only the representation, while real training also optimizes the decoder weights; the bound that makes $T>L+1$ matter also assumes a univariate representation of length $L$, which the benchmarked multivariate models exceed.

Editorial extensions

If this is right

  • If the bottleneck is real, MT-F models with $T > L+1$ have an accuracy floor that no encoder improvement can remove.
  • Pre-training for one-step prediction and adapting per segment avoids that floor while keeping the base model frozen, so long-horizon adaptation does not degrade short-horizon performance.
  • Segmenting the horizon and sharing LoRA experts through a mixture-of-experts layer keeps added parameters modest while exploiting correlations between nearby and distant steps.
  • On the reported benchmarks, MoLA improves the average MSE/MAE of iTransformer on all nine datasets and surpasses the state-of-the-art baselines on most horizons.
  • The framework also transfers to other transformer backbones (Autoformer, Informer, Transformer) and to other parameter-efficient modules (Adapter, IA3), so the decoding fix is not tied to LoRA alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theorem's 'unavoidable error' is derived by fixing $\bar{W}$ and optimizing only the augmented representation $\bar{R}$; under the actual training objective, which co-optimizes $\bar{W}$, the residual can be rotated away for a single label sequence, so the strongest form of the claim ('independent of the representation') likely does not hold for jointly trained models.
  • The rank condition is stated for a univariate representation of length $L$, while the experiments use multivariate embeddings of dimension $L\times D$ that usually exceed $T$; that suggests the empirical gains may come more from horizon-specific fine-tuning and regularization than from the theorem's bound.
  • A direct test of the mechanism: use a linear encoder so the representation has exactly $L$ features, vary $T$ across the $L+1$ threshold, and see whether the error floor predicted by the theorem appears only on the $T > L+1$ side.
  • If the bottleneck argument is what motivates horizon-specific adaptation, the same logic would apply to other readout schemes, such as probabilistic forecasters or non-linear decoders, whenever a single representation is shared across outputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that the standard multi-task forecasting (MT-F) paradigm, in which a single shared representation R is mapped to all forecast steps by a linear decoder with T outputs, suffers from an "expressiveness bottleneck": when the horizon exceeds the representation dimension plus one (T > L+1), a strictly positive minimum error is unavoidable and is independent of the encoder representation (Theorem 3.1). To avoid this bottleneck, the authors propose a two-stage framework: pre-train a model for S-step prediction (S ≤ L+1) and then adapt it to individual forecast segments using step-specific parameter-efficient modules. They instantiate this with Mixture-of-LoRA (MoLA), which shares P LoRA experts across K forecast segments through learnable segment-specific mixing weights δ_k. Experiments on ETT, ECL, Traffic, Weather, and PEMS benchmarks, using iTransformer and several backbones, show consistent improvements over the MT-F baseline and over state-of-the-art methods in most settings. The appendices include a proof of the bottleneck theorem, a variance-reduction argument for MoLA, complexity analysis, and extensive ablation results.

Significance. The paper sits at a timely intersection of parameter-efficient fine-tuning and time-series forecasting. If the expressiveness-bottleneck theorem were valid, it would provide a principled justification for replacing direct multi-step linear decoding with horizon-specific adaptation, which would be a meaningful conceptual contribution for the long-horizon forecasting community. The framework itself is simple, general, and empirically effective: MoLA consistently improves over the MT-F baseline across multiple backbones (iTransformer, Autoformer, Informer, Transformer) and datasets, with the full results table reporting 33/45 best MSE and 36/45 best MAE. The paper ships an anonymous code repository, reports ablations for K, r, η, P, layer-wise fine-tuning, and alternative PEFT modules (Adapter, IA3), and includes a parameter-count analysis showing that MoLA adds only a small fraction of parameters. These strengths make the empirical contribution credible and reproducible.

major comments (4)
  1. [Theorem 3.1 and Appendix C (Proof of Theorem C.1)] The bottleneck theorem does not analyze the MT-F model as trained. The proof fixes the decoder parameter W̄ and solves for the least-squares representation R̂ = (W̄ᵀW̄)⁻¹W̄ᵀY, reporting the residual in the null space of W̄. But in MT-F training, W̄ is co-optimized with the encoder representation R(X). For a single sequence with Y=(1,2)ᵀ, L=1, T=2, the decoder W̄=(1,2)ᵀ and R=1 give zero loss, so the claimed strictly positive lower bound when T > L+1 does not hold for the co-optimized objective. More generally, the optimal error min_{W,R} ‖Y − WR‖² depends on the representation: a degenerate encoder R=0 causes large error that no W can fix, while a rich encoder can achieve small error. The claimed independence from R is an artifact of the conditional least-squares setup and must be removed or substantially weakened.
  2. [Theorem 3.1, notation and rank bound] There is a dimensional mismatch between the theorem and the experiments. The theorem states Y ∈ R^{T×D}, R ∈ R^{L×D}, and the decoder parameter W̄ ∈ R^{T×(L+1)}, whose rank is at most min(T, L+1). The bound T > L+1 treats the representation as effectively univariate (dimension L). In the experiments, however, the encoder outputs a multivariate representation of dimension L times D (or hidden dimension d_m per variate in iTransformer), so the rank-deficient condition T > L+1 need not bind at all. As a result, the theorem's strictly positive error premise does not apply to the evaluated models. The authors should restate the theorem with the decoder as a general linear map from R^{L×D} to R^{T×D} (or make explicit the univariate assumption) and check whether the bottleneck condition can be satisfied under realistic hidden dimensions.
  3. [Appendix C, Theorem C.2] The variance-reduction theorem is circular. It assumes ΔCov(L_t, L_s) = Cov_MTF(L_t, L_s) − Cov_MoLA(L_t, L_s) ≥ 0 and then concludes Var(L_MoLA) ≤ Var(L_MTF), which is exactly the assumed inequality restated. The text explains that modularized fine-tuning introduces horizon-specific parameters and therefore reduces parameter sharing, but no mechanism is given that guarantees the covariance difference is nonnegative, and in fact adding more parameters could in principle increase variance. To be non-circular, the authors would need to bound the covariance change under the actual MoLA update rule, or explicitly label the result as an empirically observed property rather than a theorem.
  4. [Section 3.3 and Algorithm 1] The use of Theorem 3.1 to justify segment sizes S ≤ L+1 is questionable. Even if the theorem were accepted for a fixed W̄, the bound L+1 refers to the representation dimension of the linear decoder input, but the paper does not specify how L is measured in the MoLA experiments: the foundation model's intermediate representation dimension is not the lookback length L in general. In iTransformer, for instance, the token embedding dimension is d_m, not the historical window length. The condition S ≤ L+1 should be checked against the actual representation dimension of the pre-trained backbone, otherwise the claim that MoLA bypasses the bottleneck is not supported by the theorem.
minor comments (6)
  1. [Equation (1) and notation throughout] The paper uses L both for the historical window length and for the representation dimension in Theorem 3.1, and the decoder is written as W ∈ R^{T×L} multiplying R ∈ R^{L×D}, which is dimensionally inconsistent (a T×L matrix times an L×D matrix gives T×D, but the text writes W_t R with W_t apparently a row). Please clarify the exact dimensions of R, W, and the bias, and use distinct symbols for window length and representation dimension.
  2. [Section 3.2, adaptation description] The statement that the adaptation stage uses a single-output decoder and thereby eliminates the error term inherits the issues of Theorem 3.1. The empirical motivation is clear, but the sentence should be softened to say the bottleneck bound does not apply, or should defer to the corrected theorem.
  3. [Table 4 and Figure 5] The sensitivity discussion would benefit from noting that the Weather dataset shows a large discontinuous improvement at K=8 (MSE 0.173 vs 0.200–0.202 for other K values) at T=96, which may reflect lucky validation selection or a hyperparameter interaction rather than a smooth trend; a short remark would prevent over-interpretation.
  4. [Appendix B, Equation (9)] The parameter-count ratio formula appears to double-count or miscount the per-segment weight parameters: the term P×K for the mixing weights has dimensions that do not match the other terms. Please check the derivation and clarify what is being counted.
  5. [Full result tables] Several cells in Table 6 are bolded for non-MoLA entries (e.g., FreTS on Weather, DLinear on ETTm1 336) without discussion. Adding a sentence acknowledging that MoLA is not universally best would strengthen the empirical claims.
  6. [References] Reference [20] duplicates [19], and the reference list contains several formatting issues; please deduplicate and harmonize the bibliography style.

Circularity Check

2 steps flagged · score 6.0 of 10

Two theoretical planks of MoLA are circular or by construction: Theorem C.2 posits the covariance reduction it then concludes, and Theorem 3.1's 'unavoidable error independent of R' is the residual of a least-squares fit of R under a frozen decoder; the empirical benchmarks are independent and non-circular.

  1. self definitional [Appendix C, Theorem C.2 (Eq. 11) and its proof]
    "Under modularized fine-tuning, the covariances between different Lt decrease because the LoRA modules introduce horizon-specific parameters, reducing parameter sharing. Let ΔCov(Lt,Ls) = CovMT-F(Lt,Ls) − CovMoLA(Lt,Ls) ≥ 0. The variance difference is then: Var(LMT-F) − Var(LMoLA) = 2/T^2 Σ_{1≤t<s≤T} ΔCov(Lt,Ls) ≥ 0. Therefore, Var(LMoLA) = Var(LMT-F) − 2/T^2 Σ_{1≤t<s≤T} ΔCov(Lt,Ls) ≤ Var(LMT-F)."

    The theorem's conclusion, Var(LMoLA) ≤ Var(LMT-F), is logically identical to the assumed inequality ΔCov ≥ 0, which the proof introduces after merely asserting that modularized fine-tuning makes the covariances decrease. No argument is given for why the LoRA modules must reduce any covariance; the inequality is the premise, not a derived consequence. The subsequent sentence 'The equation realize only when all the LoRA modules do not contribute to the improvement of the prediction performance' confirms that the claimed variance reduction is exactly the assumption, so the theorem is equivalent to its own input.

  2. fitted input called prediction [Theorem 3.1 and Appendix C Theorem C.1, Eq. (2)/(10) and proof]
    "Theorem 3.1 (Expressiveness Bottleneck). Let ¯W = [W b] ∈ R^{T×(L+1)} be the parameters in the MT-F’s linear decoder, Y ∈ R^{T×D} be the label sequence; the minimum attainable estimation error is ∥ϵ∥^2_F = Σ_{t=rank(¯W)+1}^{T} ∥U^⊤_t Y∥^2_2 ... Notably, this error is independent of the representation R provided by encoder. Proof. Consider the least squares estimation, the aim of linear decoder is to find ¯R = [R^⊤,1]^⊤ ... such that: R̂ = arg min_{¯R} ∥Y − ¯W ¯R∥^2_2."

    The proof solves min_{¯R} ∥Y − ¯W ¯R∥^2 with ¯W frozen, so the residual is just the projection of Y onto the null space of that frozen ¯W. The named 'minimum attainable estimation error' is therefore the residual of a conditional least-squares fit of R given ¯W, not the optimum of the MT-F objective, which also trains ¯W. For a single label sequence, jointly choosing W and R can make the loss zero, so the positive lower bound and the 'independence of the representation R' are artifacts of the restricted optimization actually performed. The theorem states the residual of its own fitted R as an unavoidable property of the encoder, which is a fitted input renamed as a prediction.

full rationale

The paper has two circular or by-construction theoretical steps but no load-bearing self-citation. Theorem C.2 is circular: its proof assumes ΔCov ≥ 0 and then derives Var(LMoLA) ≤ Var(LMT-F) from that assumption, so the theorem reduces to its premise. Theorem 3.1/C.1 is also construction-dependent: the proof optimizes only ¯R while holding the decoder ¯W fixed, so the claimed 'unavoidable error independent of R' is merely the null-space residual of the frozen decoder. In the actual MT-F objective ¯W is trained too, and jointly optimizing W and R can drive the error to zero for a single sequence; thus the central expressiveness-bottleneck conclusion is an artifact of the least-squares setup rather than a property of the co-trained MT-F paradigm. The rank bound is also derived for a univariate ¯R while the paper defines Y ∈ R^{T×D}, a further correctness gap but not circularity. The empirical benchmark numbers, by contrast, are produced against external baselines under standard protocols and are not fitted to the theorem, and no central premise rests on a self-citation. Hence the partial circularity is in the theoretical justification, not in the independent experimental evaluation.

Assumptions & free parameters 4 free parameters · 6 assumptions · 2 invented entities

The central claim rests on the conditional least-squares setup of Theorem 3.1 (decoder frozen, representation optimized), on a univariate rank bound that does not transfer to the multivariate models evaluated, and on the assumed covariance reduction in Theorem C.2. The empirical claims rest on the standard benchmark protocols (chronological splits, official baseline scripts) and on per-dataset hyperparameter choices (K, P, r, η) that are tuned but not fully tabulated. No new physical or mathematical objects are invented; the LoRA experts and mixing weights are architectural parameters whose value is supported only by in-paper experiments.

free parameters (4)
  • K (number of forecast segments) = per dataset; e.g., 4 for ETTh1, 8 for Weather in headline results
    Segmentation count is chosen from {2,3,4,6} for main results and up to 96 in sensitivity tests. Performance varies strongly with K: Weather MSE 0.173 at K=8 vs 0.202 at K=2 (slightly worse than MT-F), so reported gains partly reflect the selected K.
  • P (number of LoRA experts) = tuned in {2,4,6,8,10}, per-dataset values not tabulated
    Number of shared expert matrices in the mixture; sensitivity analysis shows modest dependence unless P is far below K.
  • r (LoRA rank) = e.g., 8 optimal on ETTh1; tuned over {4,8,16,32,64}
    Rank of the low-rank adapter pair (A,B); both too-small and too-large ranks degrade performance, so the reported results depend on the chosen value.
  • eta (fine-tuning learning rate) = tuned in {0.0002,0.0005,0.001,0.002,0.005}
    Sensitivity analysis shows sharp dependence (ETTh1 MSE 0.459 at eta=0.0002 vs 0.440 at eta=0.001).
assumptions (6)
  • ad hoc to paper Decoder weights W̄ are frozen while the representation R̄ is optimized in the bottleneck analysis
    Entered at the proof of Theorem 3.1 (Appendix C). This conditional least-squares setup produces the claimed error independence from R, but MT-F training co-optimizes W̄.
  • domain assumption The representation is unconstrained and can realize the projection R̄ = (W̄ᵀW̄)⁻¹W̄ᵀY
    Required for the 'independent of encoder' conclusion; real encoders are architecture-constrained, so the claim 'regardless of encoder quality' does not follow.
  • standard math W̄ᵀW̄ is invertible
    Used in the pseudoinverse/projection step of Theorem 3.1's proof; requires full column rank of the fixed decoder.
  • ad hoc to paper Covariance reduction assumption ΔCov(Lt,Ls) ≥ 0 in Theorem C.2
    The variance-reduction theorem defines the covariance difference as non-negative and then concludes the variance is smaller; the conclusion is the assumption.
  • ad hoc to paper The evaluated models' representations have dimension L+1 as in the theorem
    Theorem 3.1's rank bound uses an L-dimensional per-variate representation, but the experiments use multivariate models (e.g., iTransformer) whose representations have dimension LD, which can exceed T, so the premise T > L+1 does not bind.
  • domain assumption Chronological train/validation/test splits and baseline reproduction scripts from the official TimesNet repository are faithful
    Standard in this benchmark suite; results hinge on these protocols but they are externally established.
invented entities (2)
  • Segment-specific mixing weights Δ_k (normalized expert-weight vectors)
    purpose: Adaptively weight the shared LoRA expert matrices for each forecast segment k in equation (6)
    No falsifiable handle outside the paper's own benchmarks; the claimed variance-reduction justification (Theorem C.2) is assumed rather than derived.
  • Shared LoRA expert matrices B^(p), A^(p)
    purpose: Partial parameter sharing across forecast segments; each segment combines the P experts via Δ_k
    Analogous to standard LoRA; the claim that inter-segment sharing helps accuracy is supported only by in-paper experiments (Table 3), and MoLA-R (no sharing) is sometimes better (Weather 720).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mixture of Low Rank Adaptation with Partial Parameter Sharing for Time Series Forecasting." pith.science (2026). https://pith.science/paper/YFTOHXM7

@misc{pith2026250517872,
  author       = {Pith},
  title        = {Pith review of: Mixture of Low Rank Adaptation with Partial Parameter Sharing for Time Series Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YFTOHXM7}},
  note         = {Machine review of arXiv:2505.17872}
}
read the original abstract

Multi-task forecasting has become the standard approach for time-series forecasting (TSF). However, we show that it suffers from an Expressiveness Bottleneck, where predictions at different time steps share the same representation, leading to unavoidable errors even with optimal representations. To address this issue, we propose a two-stage framework: first, pre-train a foundation model for one-step-ahead prediction; then, adapt it using step-specific LoRA modules.This design enables the foundation model to handle any number of forecast steps while avoiding the expressiveness bottleneck. We further introduce the Mixture-of-LoRA (MoLA) model, which employs adaptively weighted LoRA experts to achieve partial parameter sharing across steps. This approach enhances both efficiency and forecasting performance by exploiting interdependencies between forecast steps. Experiments show that MoLA significantly improves model expressiveness and outperforms state-of-the-art time-series forecasting methods. Code is available at https://anonymous.4open.science/r/MoLA-BC92.

Figures

Figures reproduced from arXiv: 2505.17872 by the authors.

Figure 1
Figure 1. Visualization of representations generated with different forecasting step. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Visualization of MT-F, LoRA and MoLA approaches to generate multi-step forecasts. Gray [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of forecast sequence generated with and without MoLA under two snapshots. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Benefit of incorporating MoLA in varying models, shown with colored bars for means over [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Performance given varying rank r, learning rate η and the number of experts P. minimal MSE values of 0.441 (192) and 0.484 (336), while higher ranks overfit and lower ranks lack expressiveness. Second, learning rate η exhibits sharper sensitivity: increasing η from 0.0…
Figure 6
Figure 6. Figure 6: Running time of MoLA given varying rank r, the number of experts P and the number of segments K. C Theoretical Justification Theorem C.1 (Expressiveness Bottleneck). Let W¯ = [W b] ∈ R T×(L+1) be the parameters in the MT-F’s linear decoder, Y ∈ R T×D be the label seque…
Figure 7
Figure 7. Figure 7: Visualization of common parameter￾efficient fine-tuning strategies. We visualize the three prominent PEFT meth￾ods: Adapter, LoRA and IA3 in [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Performance of iTransformer enhanced by MoLA given different low rank of LoRA [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Performance of iTransformer enhanced by MoLA given different low rank of LoRA [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 26 canonical work pages

  1. [1]

    End-to-end data-driven weather prediction.Nature, pages 1–3, 2025

    Anna Allen, Stratis Markou, Will Tebbutt, James Requeima, Wessel P Bruinsma, Tom R Andersson, Michael Herzog, Nicholas D Lane, Matthew Chantry, J Scott Hosking, et al. End-to-end data-driven weather prediction.Nature, pages 1–3, 2025

    work page 2025

  2. [2]

    An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.arXiv preprint arXiv:1803.01271, 2018

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.arXiv preprint arXiv:1803.01271, 2018

    arXiv 2018

  3. [3]

    Lora learns less and forgets less.arXiv preprint arXiv:2405.09673, 2024

    Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less.arXiv preprint arXiv:2405.09673, 2024

    arXiv 2024

  4. [4]

    John Wiley & Sons, 2015

    George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung.Time series analysis: forecasting and control. John Wiley & Sons, 2015

    2015

  5. [5]

    Spectral temporal graph neural network for multivariate time-series forecasting

    Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time-series forecasting. InNeurIPS, volume 33, pages 17766–17778, 2020

    work page 2020

  6. [6]

    Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms.arXiv preprint arXiv:2308.08469, 2023

    Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms.arXiv preprint arXiv:2308.08469, 2023

    arXiv 2023

  7. [7]

    Multi- scale adaptive graph neural network for multivariate time series forecasting.IEEE Transactions on Knowledge and Data Engineering, 35(10):10748–10761, 2023

    Ling Chen, Donghui Chen, Zongjiang Shang, Binqing Wu, Cen Zheng, Bo Wen, and Wei Zhang. Multi- scale adaptive graph neural network for multivariate time series forecasting.IEEE Transactions on Knowledge and Data Engineering, 35(10):10748–10761, 2023

    work page 2023

  8. [8]

    Qlora: Efficient finetuning of quantized llms.Advances in neural information processing systems, 36:10088–10115, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms.Advances in neural information processing systems, 36:10088–10115, 2023

    2023

Show all 55 references
  1. [9]

    Sparse low-rank adaptation of pre-trained language models.arXiv preprint arXiv:2311.11696, 2023

    Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models.arXiv preprint arXiv:2311.11696, 2023

    2023 arXiv

  2. [10]

    Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting

    Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting. InSIGKDD, page 459–469, 2023

    2023

  3. [11]

    A note on lora.arXiv preprint arXiv:2404.05086, 2024

    Vlad Fomenko, Han Yu, Jongho Lee, Stanley Hsieh, and Weizhu Chen. A note on lora.arXiv preprint arXiv:2404.05086, 2024

    2024 arXiv

  4. [12]

    Deep learning for time series forecasting: The electric load case.CAAI Transactions on Intelligence Technology, 7(1):1–25, 2022

    Alberto Gasparin, Slobodan Lukovic, and Cesare Alippi. Deep learning for time series forecasting: The electric load case.CAAI Transactions on Intelligence Technology, 7(1):1–25, 2022

    2022

  5. [13]

    Cross-attention is all you need: Adapting pretrained transformers for machine translation.arXiv preprint arXiv:2104.08771, 2021

    Mozhdeh Gheini, Xiang Ren, and Jonathan May. Cross-attention is all you need: Adapting pretrained transformers for machine translation.arXiv preprint arXiv:2104.08771, 2021

    2021 arXiv

  6. [14]

    Two-step deep learning framework with error compensation technique for short-term, half-hourly electricity price forecasting

    Sujan Ghimire, Ravinesh C Deo, David Casillas-Pérez, and Sancho Salcedo-Sanz. Two-step deep learning framework with error compensation technique for short-term, half-hourly electricity price forecasting. Applied Energy, 353:122059, 2024

    2024

  7. [15]

    Low-rank adaptation of time series foundational models for out-of-domain modality forecasting

    Divij Gupta, Anubhav Bhatti, Suraj Parmar, Chen Dan, Yuwei Liu, Bingjie Shen, and San Lee. Low-rank adaptation of time series foundational models for out-of-domain modality forecasting. InProceedings of the 26th International Conference on Multimodal Interaction, pages 382–386, 2024

    2024

  8. [16]

    Sensitivity-aware visual parameter- efficient fine-tuning

    Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-aware visual parameter- efficient fine-tuning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11825–11835, 2023

    2023

  9. [17]

    Towards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366, 2021

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366, 2021

    2021 arXiv

  10. [18]

    Mera: Merging pretrained adapters for few-shot learning.arXiv preprint arXiv:2308.15982, 2023

    Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. Mera: Merging pretrained adapters for few-shot learning.arXiv preprint arXiv:2308.15982, 2023

    2023 arXiv

  11. [19]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InInternational conference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  12. [20]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InInternational conference on machine learning, pages 2790–2799. PMLR, 2019. 10

    2019

  13. [21]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  14. [22]

    Fintsb: A comprehensive and practical benchmark for financial time series forecasting.arXiv preprint arXiv:2502.18834, 2025

    Yifan Hu, Yuante Li, Peiyuan Liu, Yuxia Zhu, Naiqi Li, Tao Dai, Shu-tao Xia, Dawei Cheng, and Changjun Jiang. Fintsb: A comprehensive and practical benchmark for financial time series forecasting.arXiv preprint arXiv:2502.18834, 2025

    2025 arXiv

  15. [23]

    Domain adaptation for time series transformers using one-step fine-tuning.arXiv preprint arXiv:2401.06524, 2024

    Subina Khanal, Seshu Tirupathi, Giulio Zizzo, Ambrish Rawat, and Torben Bach Pedersen. Domain adaptation for time series transformers using one-step fine-tuning.arXiv preprint arXiv:2401.06524, 2024

    2024 arXiv

  16. [24]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InICLR, 2015

    2015

  17. [25]

    Conditional adapters: Parameter-efficient transfer learning with fast inference

    Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Zhao, Yuexin Wu, Bo Li, et al. Conditional adapters: Parameter-efficient transfer learning with fast inference. Advances in Neural Information Processing Systems, 36:8152–8172, 2023

    2023

  18. [26]

    Informer: Beyond efficient transformer for long sequence time-series forecasting

    Jianxin Li, Xiong Hui, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. InAAAI, 2021

    2021

  19. [27]

    Loftq: Lora-fine-tuning-aware quantization for large language models.arXiv preprint arXiv:2310.08659, 2023

    Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-aware quantization for large language models.arXiv preprint arXiv:2310.08659, 2023

    2023 arXiv

  20. [28]

    Parameter-efficient fine-tuning without introducing new latency.arXiv preprint arXiv:2305.16742, 2023

    Baohao Liao, Yan Meng, and Christof Monz. Parameter-efficient fine-tuning without introducing new latency.arXiv preprint arXiv:2305.16742, 2023

    2023 arXiv

  21. [29]

    Sparsetsf: Modeling long-term time series forecasting with 1k parameters.arXiv preprint arXiv:2405.00946, 2024

    Shengsheng Lin, Weiwei Lin, Wentai Wu, Haojun Chen, and Junjie Yang. Sparsetsf: Modeling long-term time series forecasting with 1k parameters.arXiv preprint arXiv:2405.00946, 2024

    2024 arXiv

  22. [30]

    Segrnn: Segment recurrent neural network for long-term time series forecasting.arXiv preprint arXiv:2308.11200, 2023

    Shengsheng Lin, Weiwei Lin, Wentai Wu, Feiyu Zhao, Ruichao Mo, and Haotong Zhang. Segrnn: Segment recurrent neural network for long-term time series forecasting.arXiv preprint arXiv:2308.11200, 2023

    2023 arXiv

  23. [31]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.Advances in Neural Information Processing Systems, 35:1950–1965, 2022

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.Advances in Neural Information Processing Systems, 35:1950–1965, 2022

    1950

  24. [32]

    Scinet: time series modeling and forecasting with sample convolution and interaction

    Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: time series modeling and forecasting with sample convolution and interaction. InNeurIPS, 2022

    2022

  25. [33]

    itrans- former: Inverted transformers are effective for time series forecasting

    Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itrans- former: Inverted transformers are effective for time series forecasting. InICLR, 2024

    2024

  26. [34]

    Scaling transformer neural networks for skillful and reliable medium-range weather forecasting.Advances in Neural Information Processing Systems, 37:68740–68771, 2024

    Tung Nguyen, Rohan Shah, Hritik Bansal, Troy Arcomano, Romit Maulik, Rao Kotamarthi, Ian Foster, Sandeep Madireddy, and Aditya Grover. Scaling transformer neural networks for skillful and reliable medium-range weather forecasting.Advances in Neural Information Processing Syste...

    2024

  27. [35]

    Channel-aware low-rank adaptation in time series forecasting

    Tong Nie, Yuewen Mei, Guoyang Qin, Jian Sun, and Wei Ma. Channel-aware low-rank adaptation in time series forecasting. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 3959–3963, 2024

    2024

  28. [36]

    A time series is worth 64 words: Long-term forecasting with transformers

    Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. InICLR, 2023

    2023

  29. [37]

    Fredformer: Frequency debiased transformer for time series forecasting

    Xihao Piao, Zheng Chen, Taichi Murayama, Yasuko Matsubara, and Yasushi Sakurai. Fredformer: Frequency debiased transformer for time series forecasting. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2400–2410, 2024

    2024

  30. [38]

    Deepar: Probabilistic forecasting with autoregressive recurrent networks.Int

    David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks.Int. J. Forecast, 36(3):1181–1191, 2020

    2020

  31. [39]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, 2017

    2017

  32. [40]

    Efficient fine-tuning of bert models on the edge

    Danilo Vucetic, Mohammadreza Tayaranian, Maryam Ziaeefard, James J Clark, Brett H Meyer, and Warren J Gross. Efficient fine-tuning of bert models on the edge. In2022 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1838–1842. IEEE, 2022. 11

    2022

  33. [41]

    Micn: Multi-scale local and global context modeling for long-term series forecasting

    Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. InThe eleventh international conference on learning representations, 2023

    2023

  34. [42]

    Timesnet: Temporal 2d-variation modeling for general time series analysis

    Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. InICLR, 2023

    2023

  35. [43]

    Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting

    Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. InNeurIPS, 2021

    2021

  36. [44]

    Chain of lora: Efficient fine-tuning of language models via residual learning.arXiv preprint arXiv:2401.04151, 2024

    Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning.arXiv preprint arXiv:2401.04151, 2024

    2024 arXiv

  37. [45]

    Parameter- efficient fine-tuning for pre-trained vision models: A survey.arXiv preprint arXiv:2402.02242, 2024

    Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter- efficient fine-tuning for pre-trained vision models: A survey.arXiv preprint arXiv:2402.02242, 2024

    2024

  38. [46]

    Rethinking fourier transform from a basis functions perspective for long-term time series forecasting.Advances in Neural Information Processing Systems, 37:8515–8540, 2024

    Runze Yang, Longbing Cao, JIE YANG, et al. Rethinking fourier transform from a basis functions perspective for long-term time series forecasting.Advances in Neural Information Processing Systems, 37:8515–8540, 2024

    2024

  39. [47]

    Loretta: Low-rank economic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models.arXiv preprint arXiv:2402.11417, 2024

    Yifan Yang, Jiajun Zhou, Ngai Wong, and Zheng Zhang. Loretta: Low-rank economic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models.arXiv preprint arXiv:2402.11417, 2024

    2024 arXiv

  40. [48]

    Fouriergnn: Rethinking multivariate time series forecasting from a pure graph perspective

    Kun Yi, Qi Zhang, Wei Fan, Hui He, Liang Hu, Pengyang Wang, Ning An, Longbing Cao, and Zhendong Niu. Fouriergnn: Rethinking multivariate time series forecasting from a pure graph perspective. InNeurIPS, 2023

    2023

  41. [49]

    Frequency-domain mlps are more effective learners in time series forecasting

    Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequency-domain mlps are more effective learners in time series forecasting. In NeurIPS, 2023

    2023

  42. [50]

    Are transformers effective for time series forecasting? InAAAI, 2023

    Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? InAAAI, 2023

    2023

  43. [51]

    Solar forecasting with hourly updated numerical weather prediction.Renewable and Sustainable Energy Reviews, 154:111768, 2022

    Gang Zhang, Dazhi Yang, George Galanis, and Emmanouil Androulakis. Solar forecasting with hourly updated numerical weather prediction.Renewable and Sustainable Energy Reviews, 154:111768, 2022

    2022

  44. [52]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  45. [53]

    Film: Frequency improved legendre memory model for long-term time series forecasting.Advances in neural information processing systems, 35:12677–12690, 2022

    Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting.Advances in neural information processing systems, 35:12677–12690, 2022

    2022

  46. [54]

    FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting

    Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. InICML, 2022

    2022

  47. [55]

    One fits all: Power general time series analysis by pretrained lm.Advances in neural information processing systems, 36:43322–43355, 2023

    Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained lm.Advances in neural information processing systems, 36:43322–43355, 2023. 12 A Related Work A.1 Time Series Forecasting Modeling Time-series forecasting (TSF) m...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.