Time Series Embedding and Combination of Forecasts: A Reinforcement Learning Approach

Jeronymo M. Pinro; Marcelo C. Medeiros

REVIEW 5 major objections 5 minor 13 references

Time Series Embedding and Combination of Forecasts: A Reinforcement Learning Approach

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A reinforcement-learning agent that recalls past error profiles and selects the model that worked then can outperform simple forecast averaging in real-world panels.

desk verdict The RL selector is a nice idea, but the M4 evaluation as written is not traceable to the described algorithm, and the missing hyperparameters make the results hard to trust. read the letter →

arxiv 2508.20795 v1 pith:SXQ6GYNS submitted 2025-08-28 econ.EM

Marcelo C. Medeiros , Jeronymo M. Pinro This is my paper · ORCID

classification econ.EM

keywords forecastcombinationpuzzlereinforcementlearningdynamicmodelselectiontemporaldifferenceM4competitionSurveyofProfessionalForecastersstateembeddingaveraging

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a reinforcement-learning agent can beat the forecasting combination puzzle — the stubborn finding that simple averaging of forecasts is hard to outperform. Its proposed agent treats each forecast origin as a decision: embed the recent error profiles of all candidate models into a low-dimensional space, find the most similar past period, and choose the model that performed best then. If no past period is similar enough, it falls back to the simple average. Tested on M4 hourly data and the Survey of Professional Forecasters, the agent never wins any single forecasting contest yet ranks best overall, with an average SPF ranking of 1.93 against 3.43 for the simple average. If this holds, dynamic learning from past error histories is a credible way around equal-weight averaging in real forecast panels.

What carries the argument

The central object is a Q-table whose rows are low-dimensional PCA embeddings of cumulative squared errors and whose columns are candidate forecasting models. The mechanism is cosine-similarity retrieval of the most similar past state, adoption of that state's best model, and a temporal-difference update of the chosen model's value using the realized squared error as reward. The fallback to a simple average when similarity falls below a threshold keeps the method stable in unfamiliar regimes.

What would settle it

Rerun the M4 hourly exercise replacing the chosen 'most similar past state' with the 'least similar past state' at every forecast origin; if the realized MSE does not get worse, the cosine-similarity matching is not the source of the reported gain.

Watch

Extended reading notes

Core claim

The central claim is that forecast combination can be improved by treating model selection as a sequential decision problem and exploiting past error profiles. At each forecast origin, the agent computes cumulative squared errors of every candidate model, reduces those histories to a low-dimensional PCA embedding, and searches past embeddings by cosine similarity. If a past period is sufficiently similar, the agent copies the Q-values from that period and selects the model with the highest expected reward; otherwise it uses the simple average. The Q-table is updated with temporal-difference learning, using the realized squared error as the reward signal. Across the M4 hourly series and the S

Load-bearing premise

The method assumes that when today's error-profile embedding resembles a past period, the model that was best in that past period will also be best now; if similar histories do not transfer model superiority, the rule degrades to the simple average and the advantage disappears.

Editorial extensions

If this is right

Forecast combination can be framed as a sequential decision problem rather than a static weighting problem, and solved with standard reinforcement-learning machinery.
Practitioners with many candidate forecasts and no knowledge of the true data-generating process can apply this algorithm directly to historical error records.
The algorithm's edge appears to come from avoiding bad choices as environments change, not from being best in any single period; the average-ranking gains on both datasets support this.
The same state-embedding and similarity-retrieval procedure generalizes to any set of forecasts that leaves a track record of squared errors.
For expert panels like the SPF, the method can converge to at least the second-best expert for most series, making it a practical default in macro forecasting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the authors do not report a statistical test of the similarity-to-action transfer; a natural test is to compare the chosen model's error against that of a model chosen by a random or least-similar past state.
Editorial inference: because the state matrix is defined only by cumulative squared errors, the approach likely extends to multi-step horizons and to feature sets such as macroeconomic indicators or news counts without changing the algorithm.
Editorial inference: the hard threshold and single-winner action could be relaxed to a soft weighting over the top-k similar states, which may smooth performance when several past states tie.
Editorial inference: the gains may depend on the diversity of the forecast pool; when models are near-duplicates, the fallback simple average already solves the puzzle, consistent with the paper's results.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper proposes a Q-learning agent that picks a forecaster by embedding cumulative squared errors into a PCA space and matching the current state to the most similar past state. That is a reasonable twist on the existing RL-switching literature (Feng and Zhang 2019; Pinto and Castle 2022), and the fallback to the simple average when no similar state is found is a sensible safeguard. The authors deserve credit for framing the problem cleanly and for honestly reporting that RL is rarely the top model in individual series; they judge it by average rank.

The problems start when you look at the M4 experiment. The paper says the implemented version uses h=1, but the M4 Hourly series require 48-step-ahead forecasts from one origin. The algorithm needs a sequence of realized rewards to update the Q-table and to build past embeddings. The paper does not explain how that sequence is produced from static competition submissions. Either the authors used h=48 (contradicting the text) or they used actual future values to update the state before they would have been known. As written, the reported MSE of 15.235 is not traceable to the described workflow. This is a load-bearing problem, not a minor one, because it undercuts the claim that RL beats the competition's strong benchmarks.

The rest of the paper has smaller issues. The simple average is missing from the M4 table even though it is the paper's main benchmark. The hyperparameters—the similarity threshold eta, the number of principal components k, and the learning rate alpha—are not reported, so the method is not reproducible from the text. The SPF improvements are often tiny (e.g., 24.61 vs 24.60 for INDPROD) and no significance tests are reported; the average-rank advantage (1.93 vs 3.43) is more compelling, but we only see it in one table.

I don't think the idea should be dropped. The embedding-and-retrieval mechanism is novel enough to deserve a proper test, and the authors have at least provided a GitHub link. But the current manuscript needs a major revision: clarify how the M4 evaluation is done, report the parameters, include the simple average, and give some sense of statistical uncertainty. As is, I would not trust the empirical claims.

Who is this for? Forecasting researchers interested in adaptive model selection might want to follow the idea, but I wouldn't cite this draft. It's a paper for a serious referee, not for acceptance.

Do you want to take a look at the GitHub repo to see if the code clarifies the M4 procedure? If the code actually does something clean, that would change the picture.

Best, [Your name]

Referee Report

5 major / 5 minor

Summary. The paper proposes a reinforcement learning (RL) framework for dynamic forecast model selection, intended to outperform the simple average—the well-known forecasting combination puzzle. The method embeds a matrix of cumulative squared errors of candidate models via PCA, compares the current embedding to all past embeddings using cosine similarity, and, if the similarity exceeds a threshold, selects the model that performed best in the most similar past period; otherwise it falls back to the simple average. A Q-table is updated with realized forecast errors. The method is evaluated on M4 hourly data and on the Survey of Professional Forecasters (SPF), with the claim that RL achieves the best average ranking across experiments.

Significance. If validated, the approach would contribute to the forecast combination literature by providing a data-driven model-switching rule that adapts to changing environments. The authors make code and data publicly available and use well-known public benchmarks, which is a strength. However, the empirical evidence as presented is not sufficient to support the central claim: the M4 evaluation is inconsistent with the stated h=1 workflow, key hyperparameters are unreported, the simple-average benchmark is omitted in the M4 table, and the SPF improvements are minuscule and lack statistical inference. The idea is interesting, but the current manuscript does not yet establish that the RL method reliably beats the simple average.

major comments (5)

[§4.1 (Table 1) vs. §3.2] The M4 hourly data require 48-step-ahead forecasts from a single origin, whereas the implemented method is stated to use h=1 (Section 4). The Section 3.2 workflow requires a sequence of realized rewards G_{t+h}(a) to update the Q-table and to build cumulative squared-error embeddings. Competition submissions are static multi-horizon forecasts; the paper does not explain how one-step-ahead errors for the 61 methods were obtained over the training span. If realized test values were used to form embeddings, this introduces look-ahead; if h=48 was actually used, it contradicts the text. The reported MSE of 15.235 is therefore not traceable to the described procedure.
[§3.2] The cosine-similarity threshold η, the number of principal components k, and the learning rate α are never reported. The threshold η is central to the algorithm: it determines whether the agent switches to a past model or uses the simple average. Without these values, and without any selection criterion for them, the results are not reproducible, and the possibility of tuning to the evaluation sets cannot be assessed.
[Table 1, §4.1] The simple-average baseline is omitted from the M4 results, despite the paper's stated goal of outperforming the simple average. Table 1 lists RL and several individual competition entries but not the equal-weighted average of all 61 forecasts. Consequently, the key M4 comparison that would support the abstract and conclusion—RL beating the simple average—is not shown.
[Tables 3 and 4, §4.2] The SPF differences are extremely small: for example, COREPCE is 95.04 vs. 95.06 and HOUSING is 1333.92 for all columns. No standard errors, confidence intervals, or significance tests are provided, and the average ranking comparison (1.93 vs. 3.43) is based on only about a dozen series. The claim that RL is 'best overall choice' is not statistically supported.
[Eq. (1) vs. §3.2 update] The paper calls the method Q-learning and presents the Bellman update in Eq. (1), but the actual update in Section 3.2 is Q(St,a) ← Q(St,a) + α [G_{t+h}(a) − Q(St,a)], which omits the max over next-state actions and reduces to a TD(0) style update for the selected action only. In addition, copying Q-values from the most similar past state is not derived from the Bellman equation. The relationship between the stated RL framework and the implemented algorithm needs to be clarified.

minor comments (5)

[§1] In the introduction, 'a =, . . . , n' appears to be a typo; should read 'a = 1, . . . , n'.
[§4.2] The text says some series were excluded due to missing data, naming EMP, RCONSUM, and UNEMP. EMP is not listed in Table 2, and RCONSUM appears in Table 3. Please reconcile the list of included/excluded series.
[§4.2] The acronym RLSGOV in the text appears to be a typo for RSLGOV in Table 3.
[§3.1] The description of the matrix E_t as 'p×t' is confusing; clarify whether p is the number of features and t is the number of time periods, and explain how PCA is applied (across rows or columns) to obtain k principal components for state S_t.
[General] The paper would benefit from a reproducibility appendix listing all hyperparameter values, the initialization of the Q-table, and the handling of ties when multiple past periods have identical similarity. The GitHub repository is mentioned, but the manuscript should contain the key details.

Circularity Check

0 steps flagged · score 2.0 of 10

No material circularity; the RL evaluation is empirical and not derived from its inputs. Minor self-citation is not load-bearing.

full rationale

The paper's central claim is that a reinforcement-learning model-selection rule beats the simple average on M4 and SPF. This is an empirical claim, not a derivation. The RL algorithm is a substantive decision rule: it embeds past performance via PCA, compares current and past states by cosine similarity, selects the best past action above a threshold, and otherwise falls back to the simple average, with Q-table updates using realized rewards. The reported outperformance is not equivalent to the inputs by construction; no equation defines the output in terms of the input, and no parameter is fitted to the target result and then renamed a prediction. The only self-citation is Pinto and Castle (2022) in the Introduction, cited as prior evidence that RL can aid model selection; it is not used to justify the present method's validity or to exclude alternatives, so it is not load-bearing. A separate concern, noted in the skeptic's headline, is that Section 4 states 'h = 1' while the M4 hourly data require 48-step-ahead forecasts from a single origin; as written, the RL workflow may not be executable on that dataset. That is a correctness or reproducibility issue, not a circularity issue, because it does not make the claimed result reducible to its inputs. Accordingly, the circularity score is 2 at most, reflecting only the minor non-load-bearing self-citation; the central empirical content is independent.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim relies on the validity of the embedding-similarity transfer and on the evaluation protocol. The free parameters are undisclosed, and the domain assumptions are unproven, which weakens the paper's support.

free parameters (3)

Cosine similarity threshold eta = not reported
Controls whether to trust the matched past state; no value or tuning procedure given (Section 3.2).
Number of principal components k = not reported
Determines embedding dimension; no value or selection criterion given (Section 3.1).
Learning rate alpha = not reported
Controls Q-table update speed; no value given (Section 3.2).

assumptions (4)

domain assumption PCA on cumulative squared errors captures the relevant state for model selection.
Section 3.1 assumes that the first k principal components of E_t (which in the experiments is only cumulative squared errors) are sufficient to identify the best model.
domain assumption Similar embeddings imply similar optimal actions.
Section 3.2 selects the past period with highest cosine similarity and copies its Q-values; this assumes a Markov-like property of the embedding space.
domain assumption Online updating with actual target values during the forecast window is a valid evaluation.
In Section 4, the agent updates Q using observed outcomes at t+h, implying the test period is used for learning, which affects the comparability of the reported MSE.
domain assumption The simple average is the benchmark of the combination puzzle.
The introduction defines the combination puzzle relative to equal weighting; the evaluation therefore centers on beating the simple average.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Time Series Embedding and Combination of Forecasts: A Reinforcement Learning Approach." pith.science (2026). https://pith.science/paper/SXQ6GYNS

@misc{pith2026250820795,
  author       = {Pith},
  title        = {Pith review of: Time Series Embedding and Combination of Forecasts: A Reinforcement Learning Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SXQ6GYNS}},
  note         = {Machine review of arXiv:2508.20795}
}

read the original abstract

The forecasting combination puzzle is a well-known phenomenon in forecasting literature, stressing the challenge of outperforming the simple average when aggregating forecasts from diverse methods. This study proposes a Reinforcement Learning - based framework as a dynamic model selection approach to address this puzzle. Our framework is evaluated through extensive forecasting exercises using simulated and real data. Specifically, we analyze the M4 Competition dataset and the Survey of Professional Forecasters (SPF). This research introduces an adaptable methodology for selecting and combining forecasts under uncertainty, offering a promising advancement in resolving the forecasting combination puzzle.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

[1]

The combination of forecasts

John M Bates and Clive WJ Granger. The combination of forecasts. Journal of the operational research society, 20 0 (4): 0 451--468, 1969

work page 1969
[2]

Kaggle forecasting competitions: An overlooked learning opportunity

Casper Solheim Bojer and Jens Peder Meldgaard. Kaggle forecasting competitions: An overlooked learning opportunity. International Journal of Forecasting, 37 0 (2): 0 587--603, 2021

work page 2021
[3]

Combining forecasts: A review and annotated bibliography

Robert T Clemen. Combining forecasts: A review and annotated bibliography. International journal of forecasting, 5 0 (4): 0 559--583, 1989

work page 1989
[4]

Principled reward shaping for reinforcement learning via lyapunov stability theory

Yunlong Dong, Xiuchuan Tang, and Ye Yuan. Principled reward shaping for reinforcement learning via lyapunov stability theory. Neurocomputing, 2020

work page 2020
[5]

Crop yield prediction using deep reinforcement learning model for sustainable agrarian applications

Dhivya Elavarasan and PM Durairaj Vincent. Crop yield prediction using deep reinforcement learning model for sustainable agrarian applications. IEEE Access, 8: 0 86886--86901, 2020

work page 2020
[6]

Reinforcement learning based dynamic model selection for short-term load forecasting

Cong Feng and Jie Zhang. Reinforcement learning based dynamic model selection for short-term load forecasting. In 2019 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT), pages 1--5. IEEE, 2019

work page 2019
[7]

News Deja Vu: Connecting Past and Present with Semantic Search

Brevin Franklin, Emily Silcock, Abhishek Arora, Tom Bryan, and Melissa Dell. News deja vu: Connecting past and present with semantic search. arXiv preprint arXiv:2406.15593, 2024

work page Pith review arXiv 2024
[8]

Spatio-temporal feature fusion for dynamic taxi route recommendation via deep reinforcement learning

Shenggong Ji, Zhaoyuan Wang, Tianrui Li, and Yu Zheng. Spatio-temporal feature fusion for dynamic taxi route recommendation via deep reinforcement learning. Knowledge-Based Systems, 205: 0 106302, 2020

work page 2020

Show all 13 references

[9]

The m4 competition: 100,000 time series and 61 forecasting methods

Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36 0 (1): 0 54--74, 2020

2020
[10]

M5 accuracy competition: Results, findings, and conclusions

Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 2022

2022
[11]

Machine learning dynamic switching approach to forecasting in the presence of structural breaks

Jeronymo Marcondes Pinto and Jennifer L Castle. Machine learning dynamic switching approach to forecasting in the presence of structural breaks. Journal of Business Cycle Research, pages 1--29, 2022

2022
[12]

Reward is enough

David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence, page 103535, 2021

2021
[13]

Learning to predict by the methods of temporal differences

Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3 0 (1): 0 9--44, 1988

1988

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

The combination of forecasts

John M Bates and Clive WJ Granger. The combination of forecasts. Journal of the operational research society, 20 0 (4): 0 451--468, 1969

work page 1969

[2] [2]

Kaggle forecasting competitions: An overlooked learning opportunity

Casper Solheim Bojer and Jens Peder Meldgaard. Kaggle forecasting competitions: An overlooked learning opportunity. International Journal of Forecasting, 37 0 (2): 0 587--603, 2021

work page 2021

[3] [3]

Combining forecasts: A review and annotated bibliography

Robert T Clemen. Combining forecasts: A review and annotated bibliography. International journal of forecasting, 5 0 (4): 0 559--583, 1989

work page 1989

[4] [4]

Principled reward shaping for reinforcement learning via lyapunov stability theory

Yunlong Dong, Xiuchuan Tang, and Ye Yuan. Principled reward shaping for reinforcement learning via lyapunov stability theory. Neurocomputing, 2020

work page 2020

[5] [5]

Crop yield prediction using deep reinforcement learning model for sustainable agrarian applications

Dhivya Elavarasan and PM Durairaj Vincent. Crop yield prediction using deep reinforcement learning model for sustainable agrarian applications. IEEE Access, 8: 0 86886--86901, 2020

work page 2020

[6] [6]

Reinforcement learning based dynamic model selection for short-term load forecasting

Cong Feng and Jie Zhang. Reinforcement learning based dynamic model selection for short-term load forecasting. In 2019 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT), pages 1--5. IEEE, 2019

work page 2019

[7] [7]

News Deja Vu: Connecting Past and Present with Semantic Search

Brevin Franklin, Emily Silcock, Abhishek Arora, Tom Bryan, and Melissa Dell. News deja vu: Connecting past and present with semantic search. arXiv preprint arXiv:2406.15593, 2024

work page Pith review arXiv 2024

[8] [8]

Spatio-temporal feature fusion for dynamic taxi route recommendation via deep reinforcement learning

Shenggong Ji, Zhaoyuan Wang, Tianrui Li, and Yu Zheng. Spatio-temporal feature fusion for dynamic taxi route recommendation via deep reinforcement learning. Knowledge-Based Systems, 205: 0 106302, 2020

work page 2020