{"id":"af1e896a-34e6-4ddf-85bc-4239e3912166","arxiv_id":"2411.09928","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"For variable subset forecasting, a jointly trained imputation and forecasting framework that optimizes imputed values for forecast accuracy outperforms traditional imputation and can beat full-data forecasting.","lead":"This paper proposes a time series forecasting method that fills in missing sensor variables by optimizing the fill-in for the forecasting task rather than for accurate recovery. The method improves forecasting when only a subset of variables is available at test time, and in some cases beats the full-data baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are confounded: TOI-VSF jointly trains the forecaster on imputed inputs while baselines and the w/o-jl ablation use a forecaster trained on clean data, so the improvement may stem from distribution alignment rather than task-oriented imputation.","rationale":"I read the paper in good faith and find the motivating idea plausible: imputation for forecasting need not aim at exact recovery, and jointly training imputation with the downstream task can help. However, the experimental design does not isolate this idea. The strongest claim is the average 15% improvement over baselines and the >5% improvement over Oracle, and both comparisons are confounded. TOI-VSF retrains the forecasting model on its own imputed inputs, whereas the baselines and Oracle use a forecasting model trained on clean complete data. Table III shows this distinction matters enormously: simply attaching a pre-trained imputer to a clean-trained forecaster makes results far worse than no imputation, while joint training recovers strong performance. That pattern is exactly what one would expect if the benefit comes from aligning the forecaster's training distribution with imputed inputs, not from the imputation module producing task-oriented values. The reader's weakest_assumption identified subset-distribution generalization and a general distribution-shift concern; my concern is sharper and more load-bearing because it undermines the head-to-head baseline and Oracle comparisons even under the paper's own protocol. The w/o-jl ablation, which the paper uses to argue joint learning is essential, changes two variables at once and therefore cannot support that argument. This is addressable: rerun baselines with the same 'train forecaster on imputed data' protocol, and add a distribution-matched w/o-jl control. Since the reader's verdict is already CONDITIONAL and the proposed checks are feasible, I do not change the verdict, but I would state the condition as requiring these controlled comparisons rather than merely code release and tabulated heatmaps.","tokens_in":19756,"tokens_out":4976,"duration_ms":54516,"concrete_test":"For each imputation baseline (FDW, CSDI, SAITS, KNNE, etc.) and for TOI-VSF, train the same backbone forecaster on the imputed complete series produced by that method, giving every method the same 'joint training on imputed inputs' protocol with identical epochs and loss weights, then evaluate on the same test subsets at k=15%. If TOI-VSF still beats all baselines by a comparable margin, the task-oriented imputation claim survives; if the margins shrink or vanish, the reported 15%/5% gains are an artifact of training the forecaster on TOI-VSF's imputations. Additionally, run the missing control for Table III: pre-train the imputer with the self-supervised loss only, then train the forecaster on the pre-trained imputer's outputs with no forecasting gradient to the imputer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that task-oriented imputation outperforms recovery-based imputation by roughly 15% and beats the Oracle by over 5%—is not established by the reported comparisons because the experiments vary two factors at once: the imputation method and the training distribution of the forecasting model. In TOI-VSF (Phase II, Section III-B), the forecaster F is trained on imputed series produced by the jointly trained imputer. For the Oracle (Table II), the Partial setting (Table I), and the traditional imputation baselines (Figures 10-13), the forecasting model is trained on clean complete data and only receives imputed inputs at inference. Table III exposes the sensitivity: with a pre-trained imputer frozen and the forecaster trained on clean data ('w/o jl'), performance collapses far below Partial (e.g., ASTGCN/METR-LA MAE 9.26 vs 5.57), meaning the forecaster is not robust to imputed inputs. Joint training fixes this collapse by matching training and inference input distributions. Therefore the large gains over baselines and Oracle may be due to training the forecaster on randomly masked/imputed data (a form of augmentation or regularization), not to the imputation being 'task-oriented.' The w/o-jl ablation does not control for this: it changes both the imputation loss and the forecaster's training distribution. A fair test must hold the forecaster training protocol fixed across imputation methods.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TOI-VSF, a framework for Variable Subset Forecasting (VSF) in which a self-supervised imputation module is trained jointly with a downstream forecasting model. The imputation module reconstructs entire missing variables from a random available subset, and the total loss is a weighted combination of an imputation reconstruction loss and a forecasting loss. The authors claim that task-oriented imputation outperforms both recovery-focused imputation baselines and the Oracle setting where all variables are available at inference, reporting an average improvement of 15% over baselines and over 5% over Oracle. Experiments use four backbones (ASTGCN, MSTGCN, MTGNN, TGCN) and five datasets, with a k=15% available-variable setting in the main results.","tokens_in":20046,"tokens_out":5623,"duration_ms":56002,"significance":"The problem of forecasting when entire variables are missing at inference is practically relevant for sensor networks and IoT, and the paper's core idea—that imputation should be optimized for the downstream task rather than for accurate recovery—is plausible and worth testing. The framework is model-agnostic and the paper provides a clear description of the two-phase training procedure. A genuine positive result is that Table I shows consistent improvements of TOI-VSF over the no-imputation Partial setting across all backbone/dataset combinations, and the joint-learning idea is presented with useful ablations. However, the headline claims about superiority over recovery-based imputation and over the Oracle setting are not currently established because the experimental comparisons vary the forecaster's training distribution at the same time as the imputation loss. If the confound is resolved with controlled experiments, the paper could make a meaningful contribution; as it stands, the evidence is conditional.","major_comments":[{"comment":"The central comparison is confounded: TOI-VSF trains the forecaster F on imputed inputs generated by the jointly trained imputer (Section III-B, Eq. 13-15), whereas the Partial setting, the Oracle setting, and the traditional imputation baselines in Figures 10-13 use a forecaster trained on clean complete data and only apply imputation at inference. The 'w/o jl' ablation in Table III also changes both factors at once—it removes the forecasting loss from the imputer and returns the forecaster to clean-data training—so it does not isolate task-oriented imputation. The large gains over baselines and Oracle could therefore come from training the forecaster on randomly masked/imputed inputs (a form of augmentation or distribution matching), not from the imputation being task-oriented. To support the paper's central claim, the authors need a controlled experiment that holds the forecaster training protocol fixed across imputation methods, e.g., train the forecaster on the outputs of each imputation method in the same joint or sequential protocol, and report the resulting errors.","section":"Sections IV-B, IV-C, and Table III"},{"comment":"The abstract's headline claim of 'outperforming baseline methods by 15% on average' is not backed by any numeric comparison table. The comparisons against the imputation baselines (FDW, KNNE, IIM, TRMF, CSDI, SAITS, SS-GAN, MICE) are shown only as heatmaps in Figures 10-13, without numeric values, error bars, or standard deviations. The reader cannot verify the average improvement or its variability across the four backbones and five datasets. The paper should include a table with mean and standard deviation for every baseline, backbone, and dataset, and specify exactly how the 15% figure is computed.","section":"Abstract and Section IV-F (Figures 10-13)"},{"comment":"The claim that TOI-VSF 'consistently outperforms the Oracle setting by an average margin of over 5%' is contradicted by Table II. The per-dataset average improvements (across the four backbones) are only 1.59% and 1.71% for TRAFFIC and 3.08% and 1.34% for ETTH1 on MAE and RMSE, respectively. Thus the 'over 5%' margin does not hold for two of the five datasets, and the word 'consistently' is not supported. The claim should be restricted to the specific datasets and metrics where the margin actually holds, or the paper should lower the stated threshold.","section":"Section IV-C and Table II"},{"comment":"The conclusion in Section IV-B that TOI-VSF leads to 'consistent enhancement' and large average improvements is weakened by the absence of statistical significance tests. Several entries have overlapping error bars under a normality assumption; for example, on METR-LA with TGCN, the Partial MAE is 9.92±0.75 and the TOI-VSF MAE is 8.46±0.94, and the ETTH1 rows have standard deviations above 1.0 in many cases. Since each of the 10 runs produces paired results (same backbone and dataset), the authors should report paired significance tests (e.g., Wilcoxon signed-rank or paired t-test) or confidence intervals to support the claim that the improvements are not due to run-to-run variance.","section":"Table I"}],"minor_comments":[{"comment":"The text says the experiments use 'four real-world datasets' but then lists five datasets, including ETTH1 as item (v); the abstract also says 'four datasets.' This inconsistency should be corrected.","section":"Section IV-A"},{"comment":"Reference [27] (Hu et al., IJCAI 2024, 'Reconstructing missing variables for multivariate time series forecasting via conditional generative flows') appears to have overlapping authorship with the current paper, and it addresses a closely related problem. The paper should cite and explicitly discuss the relationship and differences between TOI-VSF and that prior work.","section":"References"},{"comment":"Equation (14) uses the notation '{xi_t:t+Q} - {~xi_t:t+Q}' for the forecasting loss, but the prediction variable was introduced as '^xi_t:t+Q' in Eq. (13); the tilde notation is inconsistent and should be replaced with the hat notation used elsewhere.","section":"Eq. (14)"},{"comment":"The ASTGCN/ETTH1 joint-learning MAE is reported as 1.87 in Table I but as 1.98 in Table III; these values should be reconciled.","section":"Tables I and III"},{"comment":"The text states that the weight-sensitivity results are 'depicted from Figure 8 to Figure 9,' but the relevant figures are Figures 5-9 across all datasets; the citation should cover all five figures.","section":"Section IV-E"},{"comment":"The training description is ambiguous: it says the subset is randomly sampled 100 times for broad coverage, but also that each epoch takes one randomly sampled variable subset. Clarify whether 100 subsets are sampled per epoch or once for the entire training run.","section":"Section IV-A Implementation Details"},{"comment":"There is a typo 'soda baseline' that should read 'SOTA baseline,' and Eq. (12) contains 'groud-truth' instead of 'ground-truth.'","section":"Section IV-A and Eq. (12)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising core idea, but the current experiments do not separate the effect of task-oriented imputation from the effect of training the forecaster on imputed inputs. The discrepancy with reference [27] from overlapping authors should also be addressed in revision; if the overlap is substantial, that may affect the novelty assessment. The heatmap-only reporting of the imputation baselines is a transparency problem that should be corrected before the paper can be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the VSF problem is worth working on, and the paper's instinct—that exact recovery is not the right target when variables vanish—is reasonable. But the headline numbers don't hold up to inspection. The 15% average improvement over baselines is computed from baselines that are shown only as heatmaps, with no table of values, no error bars, and no code. And the claim that TOI-VSF beats the Oracle by over 5% is apples-to-oranges: the Oracle forecaster is trained on clean complete data, while TOI-VSF's forecaster is trained on imputed inputs. You are varying both the imputation method and the forecaster's training distribution at once.\n\nWhat is genuinely useful: the paper assembles a clean joint-training recipe—self-supervised masked reconstruction plus forecasting loss—and shows consistent gains over the no-imputation Partial setting across four backbones and five datasets. The w/o-jl ablation is the most revealing experiment in the paper: pre-train the imputer, keep the forecaster on clean data, and performance collapses below Partial; joint training fixes it. That strongly suggests the benefit is distribution alignment (training the forecaster to accept imputed inputs), not task-oriented imputation as such. The framing of Oracle as an 'upper bound' is also wrong, since it is not an upper bound for a differently-trained forecaster.\n\nThe paper is not incoherent. The method is sensible, the writing is clear, and the motivation is decent. But the central conceptual claim—that recovery-based imputation is suboptimal and task-oriented imputation is better—is not actually tested, because no experiment holds the forecaster training protocol fixed across imputation methods. A fair comparison would train the baseline forecaster on the same imputed training distribution, or evaluate TOI-VSF with a forecaster trained on clean data. The latter exists, and it underperforms. Also worth noting: reference [27] from overlapping authors appears to be closely related; the paper cites it but doesn't discuss the relationship.\n\nWho should read this: people interested in VSF and in joint-training confounds. It deserves a serious referee, but the experiments need to be reworked. I wouldn't cite the 15% or 5% numbers as evidence until that's done.","headline":"The core claim is confounded: TOI-VSF changes both the imputation method and the forecaster's training distribution, so the 15% and Oracle-beating numbers aren't actually established.","tokens_in":20577,"tokens_out":3554,"would_cite":false,"duration_ms":35530,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that for forecasting on a partial variable set, imputation should be optimized to help the forecast, not to reconstruct the missing data accurately, and presents a joint-learning framework that does so.","keywords":["variable subset forecasting","task-oriented imputation","joint learning","self-supervised learning","multivariate time series forecasting","missing variables","spatio-temporal data","sensor failure"],"falsifier":"Train TOI-VSF as described, then evaluate on held-out subsets that are not random draws at k = 15%, such as all sensors in one contiguous road segment missing or a fixed sensor always absent. If the forecasting error on these held-out subsets is no better, or worse, than simply feeding the available subset directly to the forecasting model, the claim that task-oriented imputation generalizes across variable subsets is refuted. A second direct check is to verify whether the imputed series deviate substantially from the ground truth while predictions still improve; if no such deviation is observed, the task-oriented mechanism is not what is driving the gains.","tokens_in":19540,"feed_emoji":"🧩","tokens_out":6016,"duration_ms":58588,"temperature":0.7,"pith_summary":"Variable Subset Forecasting arises when a forecasting model trained on N variables is deployed with only S < N variables available, for example after sensor failures. The paper claims that the standard goal of imputation, faithful recovery of missing series, is the wrong objective here, and that an imputation model steered by the downstream forecasting loss yields better forecasts than both no imputation and, in several configurations, than having the complete variable set. The proposed TOI-VSF couples a self-supervised imputation module (patching, time embedding, self-attention, and TCN-based generation) with a forecasting backbone, trained jointly with a weighted sum of imputation and forecasting losses. Across four backbones and five datasets, the authors report average improvements of roughly 15% over imputation baselines and consistent gains over the Oracle setting. If correct, the work reframes missing-data handling in forecasting as task-oriented feature engineering rather than data recovery.","feed_headline":"For missing sensors, impute for the forecast, not for the truth","feed_subtitle":"On partial variable subsets, generating what helps prediction — not what matches ground truth — lifts forecasting accuracy by 15% over…","key_machinery":"The load-bearing object is the joint learning loss $L = \\alpha L_{\\mathrm{IMP}} + \\beta L_{\\mathrm{FCST}}$ with $\\alpha + \\beta = 1$, coupling a self-supervised imputation module to a forecasting model. The imputation module takes a randomly masked variable subset, patches each variable series, embeds time, applies shared multi-head self-attention over variables, and generates all N series with a two-block dilated causal TCN with residual connections; its self-supervised objective $L_{\\mathrm{IMP}}$ is mean absolute error against the ground-truth complete series. The forecasting model $F_\\Theta$ consumes the generated series and contributes $L_{\\mathrm{FCST}}$, the mean absolute error of the prediction over the lookback and horizon windows. Because gradients of $L_{\\mathrm{FCST}}$ flow back into the imputer, the imputer is pushed to produce variables that the forecaster finds useful, not variables that resemble the original data; the $L_{\\mathrm{IMP}}$ term is claimed to keep the generated series temporally coherent. This two-loss balance is what makes the imputation task-oriented.","core_discovery":"The central claim, stated on the paper's own terms, is that precise imputation is unnecessary and sometimes harmful for Variable Subset Forecasting. TOI-VSF replaces recovery-faithful imputation (minimizing error against ground-truth masks) with an imputation module that is trained to generate variables that improve the forecast of the available subset, while a self-supervised reconstruction term keeps the generated series temporally plausible. The framework is model-agnostic: it wraps existing backbones (MTGNN, ASTGCN, MSTGCN, TGCN) and is evaluated with k = 15% available variables at inference. The paper reports that TOI-VSF outperforms the Partial setting by an average of 15% and outperforms the Oracle (full-variable) setting by an average margin over 5%, arguing that joint learning lets the forecaster guide the imputer toward beneficial variables and away from redundant or negatively correlated ones. Evaluation is on five datasets (METR-LA, TRAFFIC, SOLAR, ECG5000, ETTH1) with MAE and RMSE.","pith_inferences":["Beyond the paper: because the imputer generates variables that optimize a downstream loss, the same module could be applied to other partial-input tasks such as anomaly detection or classification, where the generated variables would be shaped by task success rather than reconstruction error.","Beyond the paper: the robustness claim is established for random subset sampling at k = 15%; a natural stress test is deploying on correlated failures (for instance, a contiguous block of sensors) and checking whether the improvement persists, since such patterns are less likely to appear in the random sampler.","A manuscript-level observation rather than an inference: the abstract states four datasets, but the experimental section and tables report results on five (METR-LA, TRAFFIC, SOLAR, ECG5000, ETTH1), so the four-datasets phrase should be corrected."],"forward_implications":["If TOI-VSF is right, sensor-failure resilience in traffic, energy, and health monitoring can be achieved by learning task-oriented imputers instead of trying to reconstruct failed sensor feeds.","Because the framework is model-agnostic, any existing multivariate forecaster can be wrapped to handle partial-variable deployment without architectural changes.","The reported gains over the Oracle setting imply that on some data the complete variable set contains information that actively hurts forecasting; subsetting plus task-oriented imputation can act as a learned feature-selection mechanism.","The joint-learning ablation shows that pre-training the imputer separately and freezing it at inference loses most of the benefit, so imputation and forecasting must be co-trained.","The method maintains performance as the available subset size k decreases, suggesting graceful degradation under severe sensor loss."],"supporting_citations":[{"why":"Defines the VSF problem and provides the Forecast Distance Weighting baseline that TOI-VSF compares against.","marker":"[1]"},{"why":"Serves as the primary forecasting backbone and motivates the motivating example showing precise imputation can hurt.","marker":"[13]"},{"why":"Provides the ASTGCN and MSTGCN backbones used to test model-agnostic integration.","marker":"[18]"},{"why":"Provides the TGCN backbone used in the ablation and comparison tables.","marker":"[19]"},{"why":"CSDI is a deep diffusion imputation baseline used in RQ5 comparisons.","marker":"[11]"},{"why":"SAITS is a self-attention imputation baseline used in RQ5 comparisons.","marker":"[12]"},{"why":"TRMF is a matrix-factorization imputation baseline used in RQ5 comparisons.","marker":"[20]"},{"why":"MICE is a chained-equations imputation baseline used in RQ5 comparisons.","marker":"[22]"}],"fun_headline_variants":["Impute for forecast, not truth: 15% gain","Task-oriented imputation beats precise recovery","Don't recover missing data—optimize for prediction","For variable subsets, impute to aid forecasting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the random masked-subset sampler used during training (k = 15%, 100 random subsets per epoch) adequately represents the space of variable subsets that will appear at deployment; if a real subset pattern falls outside this sampled space, the imputer may generate misleading variables and the claimed robustness fails.","fun_headline_variants_meta":{"raw":{"variants":["Impute for forecast, not truth: 15% gain","Task-oriented imputation beats precise recovery","Don't recover missing data—optimize for prediction","For variable subsets, impute to aid forecasting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1425,"prompt_tokens":981,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":383}},"tokens_in":597,"tokens_out":444,"duration_ms":5211,"temperature":1.0,"reasoning_tokens":383,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:09:35.315329+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train TOI-VSF as described, then evaluate on held-out subsets that are not random draws at k = 15%, such as all sensors in one contiguous road segment missing or a fixed sensor always absent. If the forecasting error on these held-out subsets is no better, or worse, than simply feeding the available subset directly to the forecasting model, the claim that task-oriented imputation generalizes across variable subsets is refuted. A second direct check is to verify whether the imputed series deviate substantially from the ground truth while predictions still improve; if no such deviation is observed, the task-oriented mechanism is not what is driving the gains.","supporting_citations":[{"cited_title":"Attention based spatial- temporal graph convolutional networks for traffic flow forecasting,","cited_arxiv_id":null,"evidence_quote":"Provides the ASTGCN and MSTGCN backbones used to test model-agnostic integration."},{"cited_title":"TGCN: Time Domain Graph Convolutional Network for Multiple Objects Tracking","cited_arxiv_id":"2101.01861","evidence_quote":"Provides the TGCN backbone used in the ablation and comparison tables."},{"cited_title":"Saits: Self-attention-based imputation for time series,","cited_arxiv_id":null,"evidence_quote":"SAITS is a self-attention imputation baseline used in RQ5 comparisons."},{"cited_title":"Temporal regularized matrix factorization for high-dimensional time series prediction,","cited_arxiv_id":null,"evidence_quote":"TRMF is a matrix-factorization imputation baseline used in RQ5 comparisons."},{"cited_title":"mice: Multivariate impu- tation by chained equations in r,","cited_arxiv_id":null,"evidence_quote":"MICE is a chained-equations imputation baseline used in RQ5 comparisons."}],"review_version":1}