{"id":"cbc0a6d3-46d1-4a88-a36f-79c82955ed2d","arxiv_id":"2508.20795","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A reinforcement learning agent that matches the current forecast-error pattern to its closest historical match can select better forecasting models than simple averaging in M4 and SPF tests.","lead":"The authors propose an algorithm that learns from past forecast errors to pick the best forecasting model at each moment, instead of always averaging all forecasts. They test it on the M4 competition and U.S. professional forecast survey data, reporting that it often beats the simple average.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"M4 evaluation is inconsistent with the stated h=1: the competition provides 48-step-ahead forecasts, so the reported MSE may not come from the described RL workflow.","rationale":"The reader's weakest_assumption focused on the similarity-action transfer premise, but I identify a more basic, objective inconsistency: the evaluation protocol for M4 is incompatible with the stated algorithm. This concern directly undermines the strongest evidence for the central claim that RL outperforms the simple average. Even if the similarity-action premise holds, the M4 results cannot be validated without resolving the h=1 vs 48-step mismatch. The reader already flagged internal inconsistencies and a potentially in-sample evaluation, so my concern is related but more specific. A concrete code inspection would settle whether the reported numbers are produced by the method as described. Pending that, the central claim is unsupported, so the rejection stands.","tokens_in":5281,"tokens_out":19635,"duration_ms":178439,"concrete_test":"Inspect the GitHub repository's M4 experiment code and instrument one hourly series to log: (a) the value of h used in the forecasting loop, (b) how E_t is constructed, and (c) whether the 48 competition forecasts are treated as 1-step-ahead forecasts made at successive origins (using actuals to compute errors) or as 48-step-ahead forecasts from one origin. If h is 1 and the code advances t over the 48 test periods using observed actuals to build E_t, the method uses test-set information. If h is 48 or the code uses only the training period, it contradicts Section 4's 'h = 1'. Either outcome would show the reported M4 MSE is not produced by the described algorithm.","verdict_should_be":"REJECT","load_bearing_attack":"The central empirical claim rests on the M4 results, but the method as described cannot be applied to that data. Section 4 states that the implemented version uses h=1. Section 4.1 uses M4 hourly series, which require 48-step-ahead forecasts from a single origin. The RL workflow in Section 3.2 requires a sequence of realized rewards G_{t+h}(a) to update the Q-table and to build past embeddings from cumulative squared errors. The M4 competition submissions are static forecasts for all 48 horizons from one origin; they do not include a historical track record of each model's one-step-ahead errors over the training span. The paper does not say the authors re-fit or simulated the 61 methods to produce such errors. Thus, as written, the algorithm cannot be executed on the M4 data without either using h=48 (contradicting the text) or using realized test values to form embeddings before they would be known (look-ahead). The reported RL MSE of 15.235 and the claim that RL beats strong benchmarks are therefore not traceable to the described procedure. The omission of the simple-average baseline in Table 1 compounds the issue, since the paper's stated goal is to beat that benchmark.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a reinforcement learning (RL) framework for dynamic forecast model selection, intended to outperform the simple average—the well-known forecasting combination puzzle. The method embeds a matrix of cumulative squared errors of candidate models via PCA, compares the current embedding to all past embeddings using cosine similarity, and, if the similarity exceeds a threshold, selects the model that performed best in the most similar past period; otherwise it falls back to the simple average. A Q-table is updated with realized forecast errors. The method is evaluated on M4 hourly data and on the Survey of Professional Forecasters (SPF), with the claim that RL achieves the best average ranking across experiments.","tokens_in":5602,"tokens_out":5522,"duration_ms":55411,"significance":"If validated, the approach would contribute to the forecast combination literature by providing a data-driven model-switching rule that adapts to changing environments. The authors make code and data publicly available and use well-known public benchmarks, which is a strength. However, the empirical evidence as presented is not sufficient to support the central claim: the M4 evaluation is inconsistent with the stated h=1 workflow, key hyperparameters are unreported, the simple-average benchmark is omitted in the M4 table, and the SPF improvements are minuscule and lack statistical inference. The idea is interesting, but the current manuscript does not yet establish that the RL method reliably beats the simple average.","major_comments":[{"comment":"The M4 hourly data require 48-step-ahead forecasts from a single origin, whereas the implemented method is stated to use h=1 (Section 4). The Section 3.2 workflow requires a sequence of realized rewards G_{t+h}(a) to update the Q-table and to build cumulative squared-error embeddings. Competition submissions are static multi-horizon forecasts; the paper does not explain how one-step-ahead errors for the 61 methods were obtained over the training span. If realized test values were used to form embeddings, this introduces look-ahead; if h=48 was actually used, it contradicts the text. The reported MSE of 15.235 is therefore not traceable to the described procedure.","section":"§4.1 (Table 1) vs. §3.2"},{"comment":"The cosine-similarity threshold η, the number of principal components k, and the learning rate α are never reported. The threshold η is central to the algorithm: it determines whether the agent switches to a past model or uses the simple average. Without these values, and without any selection criterion for them, the results are not reproducible, and the possibility of tuning to the evaluation sets cannot be assessed.","section":"§3.2"},{"comment":"The simple-average baseline is omitted from the M4 results, despite the paper's stated goal of outperforming the simple average. Table 1 lists RL and several individual competition entries but not the equal-weighted average of all 61 forecasts. Consequently, the key M4 comparison that would support the abstract and conclusion—RL beating the simple average—is not shown.","section":"Table 1, §4.1"},{"comment":"The SPF differences are extremely small: for example, COREPCE is 95.04 vs. 95.06 and HOUSING is 1333.92 for all columns. No standard errors, confidence intervals, or significance tests are provided, and the average ranking comparison (1.93 vs. 3.43) is based on only about a dozen series. The claim that RL is 'best overall choice' is not statistically supported.","section":"Tables 3 and 4, §4.2"},{"comment":"The paper calls the method Q-learning and presents the Bellman update in Eq. (1), but the actual update in Section 3.2 is Q(St,a) ← Q(St,a) + α [G_{t+h}(a) − Q(St,a)], which omits the max over next-state actions and reduces to a TD(0) style update for the selected action only. In addition, copying Q-values from the most similar past state is not derived from the Bellman equation. The relationship between the stated RL framework and the implemented algorithm needs to be clarified.","section":"Eq. (1) vs. §3.2 update"}],"minor_comments":[{"comment":"In the introduction, 'a =, . . . , n' appears to be a typo; should read 'a = 1, . . . , n'.","section":"§1"},{"comment":"The text says some series were excluded due to missing data, naming EMP, RCONSUM, and UNEMP. EMP is not listed in Table 2, and RCONSUM appears in Table 3. Please reconcile the list of included/excluded series.","section":"§4.2"},{"comment":"The acronym RLSGOV in the text appears to be a typo for RSLGOV in Table 3.","section":"§4.2"},{"comment":"The description of the matrix E_t as 'p×t' is confusing; clarify whether p is the number of features and t is the number of time periods, and explain how PCA is applied (across rows or columns) to obtain k principal components for state S_t.","section":"§3.1"},{"comment":"The paper would benefit from a reproducibility appendix listing all hyperparameter values, the initialization of the Q-table, and the handling of ties when multiple past periods have identical similarity. The GitHub repository is mentioned, but the manuscript should contain the key details.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The M4 protocol inconsistency is the most serious issue and should be addressed head-on. If the authors can clarify or re-run the M4 evaluation in a way consistent with the described method (or clearly adopt h=48), and if they report all hyperparameters and include the simple-average benchmark, the paper may be viable after revision. Otherwise, the empirical contribution is not credible. The SPF results need formal inference before any claim of superiority can be made."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper proposes a Q-learning agent that picks a forecaster by embedding cumulative squared errors into a PCA space and matching the current state to the most similar past state. That is a reasonable twist on the existing RL-switching literature (Feng and Zhang 2019; Pinto and Castle 2022), and the fallback to the simple average when no similar state is found is a sensible safeguard. The authors deserve credit for framing the problem cleanly and for honestly reporting that RL is rarely the top model in individual series; they judge it by average rank.\n\nThe problems start when you look at the M4 experiment. The paper says the implemented version uses h=1, but the M4 Hourly series require 48-step-ahead forecasts from one origin. The algorithm needs a sequence of realized rewards to update the Q-table and to build past embeddings. The paper does not explain how that sequence is produced from static competition submissions. Either the authors used h=48 (contradicting the text) or they used actual future values to update the state before they would have been known. As written, the reported MSE of 15.235 is not traceable to the described workflow. This is a load-bearing problem, not a minor one, because it undercuts the claim that RL beats the competition's strong benchmarks.\n\nThe rest of the paper has smaller issues. The simple average is missing from the M4 table even though it is the paper's main benchmark. The hyperparameters—the similarity threshold eta, the number of principal components k, and the learning rate alpha—are not reported, so the method is not reproducible from the text. The SPF improvements are often tiny (e.g., 24.61 vs 24.60 for INDPROD) and no significance tests are reported; the average-rank advantage (1.93 vs 3.43) is more compelling, but we only see it in one table.\n\nI don't think the idea should be dropped. The embedding-and-retrieval mechanism is novel enough to deserve a proper test, and the authors have at least provided a GitHub link. But the current manuscript needs a major revision: clarify how the M4 evaluation is done, report the parameters, include the simple average, and give some sense of statistical uncertainty. As is, I would not trust the empirical claims.\n\nWho is this for? Forecasting researchers interested in adaptive model selection might want to follow the idea, but I wouldn't cite this draft. It's a paper for a serious referee, not for acceptance.\n\nDo you want to take a look at the GitHub repo to see if the code clarifies the M4 procedure? If the code actually does something clean, that would change the picture.\n\nBest,\n[Your name]","headline":"The RL selector is a nice idea, but the M4 evaluation as written is not traceable to the described algorithm, and the missing hyperparameters make the results hard to trust.","tokens_in":6026,"tokens_out":3286,"would_cite":false,"duration_ms":28924,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reinforcement-learning agent that recalls past error profiles and selects the model that worked then can outperform simple forecast averaging in real-world panels.","keywords":["forecast combination puzzle","reinforcement learning","dynamic model selection","temporal difference learning","M4 competition","Survey of Professional Forecasters","state embedding","forecast averaging"],"falsifier":"Rerun the M4 hourly exercise replacing the chosen 'most similar past state' with the 'least similar past state' at every forecast origin; if the realized MSE does not get worse, the cosine-similarity matching is not the source of the reported gain.","tokens_in":5230,"feed_emoji":"📈","tokens_out":8233,"duration_ms":78615,"temperature":0.7,"pith_summary":"The paper sets out to show that a reinforcement-learning agent can beat the forecasting combination puzzle — the stubborn finding that simple averaging of forecasts is hard to outperform. Its proposed agent treats each forecast origin as a decision: embed the recent error profiles of all candidate models into a low-dimensional space, find the most similar past period, and choose the model that performed best then. If no past period is similar enough, it falls back to the simple average. Tested on M4 hourly data and the Survey of Professional Forecasters, the agent never wins any single forecasting contest yet ranks best overall, with an average SPF ranking of 1.93 against 3.43 for the simple average. If this holds, dynamic learning from past error histories is a credible way around equal-weight averaging in real forecast panels.","feed_headline":"Reinforcement learning beats the forecast-average puzzle","feed_subtitle":"A model-selection agent beat simple averages and industry experts on M4 and SPF — without winning any single contest.","key_machinery":"The central object is a Q-table whose rows are low-dimensional PCA embeddings of cumulative squared errors and whose columns are candidate forecasting models. The mechanism is cosine-similarity retrieval of the most similar past state, adoption of that state's best model, and a temporal-difference update of the chosen model's value using the realized squared error as reward. The fallback to a simple average when similarity falls below a threshold keeps the method stable in unfamiliar regimes.","core_discovery":"The central claim is that forecast combination can be improved by treating model selection as a sequential decision problem and exploiting past error profiles. At each forecast origin, the agent computes cumulative squared errors of every candidate model, reduces those histories to a low-dimensional PCA embedding, and searches past embeddings by cosine similarity. If a past period is sufficiently similar, the agent copies the Q-values from that period and selects the model with the highest expected reward; otherwise it uses the simple average. The Q-table is updated with temporal-difference learning, using the realized squared error as the reward signal. Across the M4 hourly series and the S","pith_inferences":["Editorial inference: the authors do not report a statistical test of the similarity-to-action transfer; a natural test is to compare the chosen model's error against that of a model chosen by a random or least-similar past state.","Editorial inference: because the state matrix is defined only by cumulative squared errors, the approach likely extends to multi-step horizons and to feature sets such as macroeconomic indicators or news counts without changing the algorithm.","Editorial inference: the hard threshold and single-winner action could be relaxed to a soft weighting over the top-k similar states, which may smooth performance when several past states tie.","Editorial inference: the gains may depend on the diversity of the forecast pool; when models are near-duplicates, the fallback simple average already solves the puzzle, consistent with the paper's results."],"forward_implications":["Forecast combination can be framed as a sequential decision problem rather than a static weighting problem, and solved with standard reinforcement-learning machinery.","Practitioners with many candidate forecasts and no knowledge of the true data-generating process can apply this algorithm directly to historical error records.","The algorithm's edge appears to come from avoiding bad choices as environments change, not from being best in any single period; the average-ranking gains on both datasets support this.","The same state-embedding and similarity-retrieval procedure generalizes to any set of forecasts that leaves a track record of squared errors.","For expert panels like the SPF, the method can converge to at least the second-best expert for most series, making it a practical default in macro forecasting."],"supporting_citations":[{"why":"Defines the forecast combination problem and the question of whether combining forecasts can beat the simple average.","marker":"Bates and Granger [1969]"},{"why":"Documents that equal weighting of forecasts is frequently the most effective approach, setting up the forecasting combination puzzle.","marker":"Clemen [1989]"},{"why":"Supplies the M4 competition dataset with 61 forecasting methods that serve as the action set for the RL agent.","marker":"Makridakis et al. [2020]"},{"why":"Shows that reinforcement learning can be used for dynamic model selection in short-term load forecasting, providing the direct precedent.","marker":"Feng and Zhang [2019]"},{"why":"Provides evidence that machine-learning switching approaches, including RL, adapt to structural breaks in forecasting.","marker":"Pinto and Castle [2022]"},{"why":"Supplies the temporal-difference learning rule that underlies the Q-table update used in the paper's algorithm.","marker":"Sutton [1988]"},{"why":"Motivates the semantic-search-like step of retrieving the most similar past state, which is the core similarity mechanism.","marker":"Franklin et al. [2024]"}],"fun_headline_variants":["RL model choice beats simple forecast averages","Reinforcement learning outdoes naive forecast combo","Forecast agent wins via embedding, not single wins","RL forecast selection tops average without top pick","Time-series embeddings steer RL to better forecast mix"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The method assumes that when today's error-profile embedding resembles a past period, the model that was best in that past period will also be best now; if similar histories do not transfer model superiority, the rule degrades to the simple average and the advantage disappears.","fun_headline_variants_meta":{"raw":{"variants":["RL model choice beats simple forecast averages","Reinforcement learning outdoes naive forecast combo","Forecast agent wins via embedding, not single wins","RL forecast selection tops average without top pick","Time-series embeddings steer RL to better forecast mix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1172,"prompt_tokens":590,"completion_tokens":582,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":334,"completion_tokens_details":{"reasoning_tokens":523}},"tokens_in":334,"tokens_out":582,"duration_ms":6517,"temperature":1.0,"reasoning_tokens":523,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:47:43.730603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the M4 hourly exercise replacing the chosen 'most similar past state' with the 'least similar past state' at every forecast origin; if the realized MSE does not get worse, the cosine-similarity matching is not the source of the reported gain.","supporting_citations":[{"cited_title":"The combination of forecasts","cited_arxiv_id":null,"evidence_quote":"Defines the forecast combination problem and the question of whether combining forecasts can beat the simple average."},{"cited_title":"Combining forecasts: A review and annotated bibliography","cited_arxiv_id":null,"evidence_quote":"Documents that equal weighting of forecasts is frequently the most effective approach, setting up the forecasting combination puzzle."},{"cited_title":"The m4 competition: 100,000 time series and 61 forecasting methods","cited_arxiv_id":null,"evidence_quote":"Supplies the M4 competition dataset with 61 forecasting methods that serve as the action set for the RL agent."},{"cited_title":"Reinforcement learning based dynamic model selection for short-term load forecasting","cited_arxiv_id":null,"evidence_quote":"Shows that reinforcement learning can be used for dynamic model selection in short-term load forecasting, providing the direct precedent."},{"cited_title":"Machine learning dynamic switching approach to forecasting in the presence of structural breaks","cited_arxiv_id":null,"evidence_quote":"Provides evidence that machine-learning switching approaches, including RL, adapt to structural breaks in forecasting."},{"cited_title":"News Deja Vu: Connecting Past and Present with Semantic Search","cited_arxiv_id":"2406.15593","evidence_quote":"Motivates the semantic-search-like step of retrieving the most similar past state, which is the core similarity mechanism."}],"review_version":1}