{"id":"f88d1277-7dec-4f5e-94e8-e51901195fa0","arxiv_id":"2412.00108","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Act-Now combines random subgraph sampling, fast and slow stream buffers, and a label-decomposition forecaster, reporting large MSE reductions on three cellular traffic datasets.","lead":"This paper introduces Act-Now, an online forecasting framework for large-scale streaming data that updates models with partial labels, pseudo-labels, and delayed full labels while sampling random subgraphs to fit on GPUs. The authors report average error reductions of 28.4% and 19.5% on two cellular traffic datasets, though key assumptions and novelty need scrutiny.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I contradicts the SOTA claim and the headline 28.4%/19.5% improvements are not reproducible from the reported numbers.","rationale":"The central claim of the paper is the quantitative superiority of Act-Now/Lade, summarized as 'average 28.4% and 19.5% performance improvement' and 'Lade consistently achieves SOTA.' These statements appear in the abstract, introduction, and Section VI-D. The paper's own Table I directly contradicts the SOTA claim: on C2TM with input-8-output-4, Lade's MSE is 1.984 versus Offline's 1.974. The paper itself admits this in Section VI-D ('Lade performs worse than the offline model on the input-8-output-4 setup'), making the earlier unconditional SOTA claim false as written. The reported average improvements are also not reproducible from Table I by averaging across baselines, horizons, or datasets; no straightforward aggregation gives 28.4% for CBS and 19.5% for Milano. This is more load-bearing than the pseudo-label concern because it attacks the headline result directly rather than a component's generalization. The pseudo-label assumption (Proposition 1) is real but less central: FSB contributes only ~1.4% in the Milano ablation, and even if its assumption were false, the framework could still be valuable. The reader's verdict of CONDITIONAL remains appropriate: these issues are correctable in revision (reword the SOTA claim, define the aggregation, provide raw results), and the underlying framework may still perform well if independently reproduced. I therefore do not recommend changing the verdict, but I do recommend that the authors supply the exact computation behind the 28.4%/19.5% figures and reconcile the C2TM contradiction.","tokens_in":17107,"tokens_out":12430,"duration_ms":105989,"concrete_test":"Reproduce Table I from the released code (github.com/Anoise/Act-Now) using the reported settings, then recompute the relative MSE reductions with the authors' exact aggregation. If the reproduction confirms Offline 1.974 vs Lade 1.984 on C2TM length 4, the SOTA claim is false as stated; if the headline 28.4%/19.5% figures match no well-defined aggregation of the reproduced table, they must be corrected or removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's own Table I falsifies the central SOTA claim. In Section VI-D, the authors state 'Lade consistently achieves state-of-the-art (SOTA) performance across all datasets and prediction length settings,' but the same section reports that on C2TM with input-8-output-4, Lade (MSE 1.984) is worse than Offline (MSE 1.974). The abstract and introduction repeat this SOTA/improvement claim. Additionally, the abstract's 'average 28.4% and 19.5% performance improvement' cannot be derived from Table I by any standard aggregation: e.g., on CBS, Lade vs ER gives 14.5%/9.6%/9.0% reductions (avg ~11.0%), and vs MIR gives ~13.9%/9.0%/8.3% (avg ~10.4%); on Milano, vs ER gives ~9.6%/11.2%/31.9% (avg ~17.6%) and vs MIR ~8.9%/7.4%/2.2% (avg ~6.2%). The closest per-dataset average to 28.4% is Lade vs DER++ on CBS (~27.5%); no subset yields 19.5% for Milano. Thus the main quantitative claims are unsupported by the paper's own evidence. The pseudo-label assumption of Proposition 1 is also unproved, but it is a secondary concern: the FSB ablation contributes only ~1.4% on Milano (Fig. 6b), whereas the headline numbers themselves are the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Act-Now, an online forecasting framework for large-scale streaming data. It introduces Random Subgraph Sampling (RSS) to partition large graphs into GPU-sized subgraphs, a Fast Stream Buffer (FSB) that updates the model with partial labels and consistent pseudo-labels, a Slow Stream Buffer (SSB) that updates the model in parallel with earlier full labels, online updates on the validation set, and a Label Decomposition model (Lade) with statistical and normalization flows. The central claims are that Lade achieves state-of-the-art performance across all datasets and prediction lengths and that the framework yields average MSE improvements of 28.4% and 19.5% over existing methods. Experiments are reported on the CBS, Milano, and C2TM datasets, and a GitHub repository is provided for reproducibility.","tokens_in":17485,"tokens_out":8108,"duration_ms":67533,"significance":"If the claims held, Act-Now would be a useful contribution: it addresses a genuine issue in online forecasting (information leakage in model updates), proposes a GPU-feasible sampling strategy for large graph-structured streams, and provides an ablation study plus an open-source implementation. The Lade architecture and the dual-buffer design are interesting. However, the headline quantitative claims are not supported by the paper's own Table I, the consistent-SOTA statement is contradicted by the C2TM input-8-predict-4 result, and the FSB pseudo-label mechanism rests on an unproved superiority assumption. The framework is potentially valuable, but the evidence as presented requires correction and more careful evaluation before the stated conclusions can be accepted.","major_comments":[{"comment":"The claim that Lade 'consistently achieves state-of-the-art (SOTA) performance across all datasets and prediction length settings' is contradicted by Table I: for C2TM with input length 8 and prediction length 4, Lade reports MSE 1.984, while Offline and OnlineTCN report 1.974, and ER, MIR, and DER++ report 1.976. The text partly acknowledges this in Section VI-D ('Lade performs worse than the offline model on the input-8-output-4 setup'), but the unqualified SOTA statement appears in the same section and in the abstract. The SOTA claim must be corrected to reflect the actual configuration.","section":"Abstract; Section VI-D; Table I"},{"comment":"The abstract and Section VI-D report average improvements of 28.4% and 19.5% 'compared to the advanced ER and MIR' on CBS and Milano, respectively. The numbers in Table I do not support this aggregation. For CBS, the summed MSE over prediction lengths 24, 48, and 72 is 9.105 for Lade, 10.205 for ER (a 10.8% reduction), and 10.136 for MIR (10.2%). For Milano, the summed MSE is 6.297 for Lade, 7.843 for ER (19.7%, close to the stated 19.5%), but 6.699 for MIR (6.0%). No standard aggregation over the reported rows yields 28.4% for CBS. The authors must either state the exact formula used (e.g., which baselines are averaged, whether horizons are pooled) or revise the headline claims.","section":"Abstract; Section VI-D; Table I"},{"comment":"The consistent pseudo-label loss trains the model to match its own newer predictions, so the only stated justification is Proposition 1, which asserts that newer-input predictions are superior without proof or measurement. The distance metric in Eq. (2) is also not defined. Because this loss is part of the FSB contribution, the FSB ablation results in Fig. 6(b) are difficult to interpret: under abrupt concept drift the pseudo-label could be biased and the loss could reinforce errors. The full-label loss Eq. (10) and SSB provide independent supervision, so the framework is not circular overall, but the value of the FSB component needs a test that compares against partial-label-only updates or injects known drift to check pseudo-label quality.","section":"Section IV-D, Eq. (2)"},{"comment":"Table III reports an offline Lade MSE of 1.841 for Milano at prediction length 24, while Table I reports the offline MSE on the same setting as 2.330, and Table II reports values of 2.268, 2.330, and 2.357 depending on the number of partitions. The online Lade value in Table III (1.802) also differs from Table I (1.797). The paper does not explain these discrepancies, which makes the performance-improvement percentages in Table III hard to verify. Please clarify the experimental setup or use one consistent set of results.","section":"Table III vs Table I"}],"minor_comments":[{"comment":"The text uses 'SSB+FFB+Val' where the second component should be 'FSB'; the acronym FFB is not introduced anywhere.","section":"Section VI-F"},{"comment":"The text uses 'input-8-output-4' while the dataset description uses 'Input-8-predict-4'; please use one notation throughout the paper.","section":"Section VI-D"},{"comment":"In the proof of Theorem 1, Eq. (12) writes the sampled neighborhood as N(v) \\cup V', but the expectation step uses only N(v); the sampling probability P(u) is not defined in terms of Algorithm 1. Please rewrite the proof with explicit notation for the sampled set and the probability model.","section":"Appendix B"},{"comment":"The value of the constant epsilon is not given in Section VI-B, and Var(Y', -1) does not specify the axis along which the variance is computed; please state these details.","section":"Eqs. (5)-(6)"},{"comment":"The column labeled 'Inference (S/Epoch)' is misleading because inference is not measured per epoch; rename it (for example, 'inference time per pass') and state the batch size.","section":"Section VI-J, Table IV"},{"comment":"Figure 6(b) reports ablation results without error bars, even though the text says all methods are run at least three times; please add error bars or state why they are omitted.","section":"Section VI-E"}],"recommendation":"major_revision","confidential_remarks":"The paper's central quantitative claims are overstated relative to its own table, so the manuscript needs substantial revision before it can be considered for publication. I also suggest asking the authors to discuss the relationship to their prior PSLD work (Ref. [46]): Lade appears to be a direct extension of the label-decomposition idea from that paper, and the novelty boundary of Act-Now relative to PSLD should be stated explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a real core: an online forecasting framework that respects delayed labels, uses random subgraph sampling to fit on a GPU, and splits updates into a fast buffer (pseudo/partial labels) and a slow buffer (full labels). The incremental buffer that stores each time point once is a nice engineering detail, and the ablations in Fig. 6 suggest each component helps. The unbiasedness proof for RSS is elementary but correct. That part is worth taking seriously.\n\nThe problem is the packaging. The abstract and Section VI-D claim an average 28.4%/19.5% improvement over ER/MIR and that 'Lade consistently achieves SOTA across all datasets and prediction lengths.' Table I doesn't support either claim. On C2TM input-8-predict-4, Lade (1.984) is worse than Offline (1.974). Recomputing the averages from Table I, CBS vs ER gives about 11% rather than 28.4%; Milano vs ER about 17.6%, not 19.5%. The closest to 28.4% is Lade vs DER++ on CBS, but the text says ER and MIR. So the headline numbers are wrong, and this is load-bearing because that's the paper's main selling point.\n\nThere are two more soft spots. Proposition 1 assumes newer predictions are better than older ones so they can serve as pseudo-labels. That's asserted, not demonstrated. Given the FSB ablation on Milano is only ~1.4%, it's not a fatal flaw, but it should be tested. And Lade is structurally close to the authors' own PSLD (which appears as a baseline in Table III) without stating the relationship. That needs to be disclosed.\n\nMinor but relevant: no error bars or seed information, and the practice of updating on the validation set during the stream means the validation set is no longer a held-out selection set. The paper should explain exactly how evaluation is kept honest.\n\nIf the authors fix the numbers, disclose the PSLD lineage, and add basic uncertainty reporting, this becomes a useful engineering contribution for large-scale streaming forecasting. I'd send it to referees, but with a clear request for major revision on the claims.","headline":"Useful online-forecasting framework, but the paper's headline numbers and SOTA claim don't survive contact with its own Table I.","tokens_in":18007,"tokens_out":3889,"would_cite":false,"duration_ms":32110,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Act-Now claims that large-scale streaming forecasting can be made causal and accurate by updating with consistent pseudo-labels, and reports large error reductions over continual-learning baselines.","keywords":["online forecasting","streaming data","concept drift","random subgraph sampling","label decomposition","continual learning","spatio-temporal forecasting","information leakage"],"falsifier":"Take a stream with sudden, repeated regime shifts, and compare Act-Now with FSB enabled versus disabled: if MSE and pseudo-label error rise together when drift is abrupt, Proposition 1 fails. More directly, at each FSB update, check whether the newer prediction's error is actually lower than the older prediction's on the eventual ground truth; if it is not, the pseudo-label target is misleading.","tokens_in":16907,"feed_emoji":"📈","tokens_out":4963,"duration_ms":42502,"temperature":0.7,"pith_summary":"This paper tries to establish that online forecasting of very large streaming data can be done both causally and accurately. Existing online methods update models with future labels, leaking information; when that leakage is removed, accuracy collapses. The proposed Act-Now framework instead keeps the model learning immediately with partial labels plus consistent pseudo-labels, and separately uses complete labels from earlier moments, so nothing future is seen. On three large real-world city-scale datasets, the label-decomposition model Lade reports lower MSE than the continual-learning baselines, with average 28.4% and 19.5% relative reductions on the CBS and Milano datasets. A sympathetic reader would care because this is a path to GPU-feasible, causality-respecting online forecasting at city scale.","feed_headline":"Online forecasting cuts error 28% without peeking at the future","feed_subtitle":"New framework updates models with pseudo-labels and full labels, so city-scale streams can learn causally on one GPU.","key_machinery":"The central mechanism is a split-stream update scheme. A Fast Stream Buffer (FSB) updates the model immediately using partial true labels and 'consistent pseudo-labels'—the prediction made from recent inputs is treated as a target for the older prediction via Eq. (2), on the assumption (Proposition 1) that newer predictions are better. A Slow Stream Buffer (SSB) updates the model in parallel on other GPUs with complete labels from earlier time steps, and the validation set is itself updated online. To resist concept drift, the Lade model decomposes each target into a mean, variance, and normalized residual flow (mean-variance decomposition), predicts each component with separate predictors, and recombines them through a learnable combiner; the statistical flow and normalization flow are trained by separate optimizers. Random Subgraph Sampling (RSS) cuts the large graph into random subgraphs, with Theorem 1 stating that the sampled subgraph's aggregated features are an unbiased estimate of the full graph's features.","core_discovery":"The paper's central claim is that its Act-Now framework—random subgraph sampling, fast and slow stream buffers, the Lade label-decomposition model, and online updates on the validation set—makes large-scale streaming forecasting both leakage-free and state of the art. On the CBS, Milano, and C2TM datasets, Lade is reported to achieve the lowest MSE/MAE across all prediction lengths compared with Experience Replay, MIR, FSNet, OnlineTCN, DER++, and offline training. The reported headline numbers are a 28.4% average relative cumulative MSE reduction over ER and MIR on CBS and 19.5% on Milano. The framework also claims versatility: wrapping other forecasting models in it improves them, for example Informer by 56% and 41% at horizons 24 and 48.","pith_inferences":["If consistent pseudo-labels work because nearby predictions are more accurate, the same trick could apply to any delayed-label setting—recommender systems, sensor fusion, or reinforcement learning—where an immediate target is needed before ground truth arrives.","The claimed gains likely depend on stream smoothness: under abrupt regime changes, newer predictions may be biased, so a change-point detector or confidence gate on pseudo-labels would be a natural extension the paper does not test.","RSS's unbiasedness result suggests the approach could scale beyond the tested tens of thousands of nodes to much larger networks, or be combined with variational node sampling to control which parts of the graph get updated most often.","Lade's variance component is trained as a target, not just a statistic, which may give prediction-interval estimates for free; the paper notes this at the end but does not quantify coverage."],"forward_implications":["Removing information leakage no longer has to cost accuracy: immediate updates can come from partial labels and consistent pseudo-labels while full labels arrive later.","Large graph-structured streams can be trained on a single GPU by repeatedly sampling subgraphs, since RSS is claimed to be an unbiased estimator of full-graph features.","Validation data can be folded into online training, so the model keeps learning continuously instead of freezing after model selection.","The framework is modular: other forecasting backbones plugged into Act-Now improve, often by double-digit percentages, so the benefits are not tied to Lade alone.","Lade's separate statistical and normalization flows give faster training and inference than transformer baselines while keeping lower MSE."],"supporting_citations":[{"why":"Defines the FSNet baseline whose fast and slow learning is compared and is said to become ineffective once information leakage is removed.","marker":"[15]"},{"why":"Provides the Experience Replay baseline that Act-Now is compared against for cumulative MSE on the large-scale datasets.","marker":"[31]"},{"why":"Supplies the tiny episodic memory variant of experience replay used as a baseline.","marker":"[43]"},{"why":"Defines MIR, the continual-learning baseline that Lade's average 19.5% MSE reduction on Milano is measured against.","marker":"[44]"},{"why":"Defines DER++, a dark experience replay baseline used in the online forecasting comparison.","marker":"[45]"},{"why":"Provides the OnlineTCN baseline based on online convex programming.","marker":"[42]"},{"why":"Defines concept drift, the problem the Lade model and the online update scheme are designed to handle.","marker":"[22]"},{"why":"Supplies the Milano dataset, one of the three large-scale streaming datasets used for evaluation.","marker":"[39]"},{"why":"Supplies the C2TM dataset, the short, high-noise stream used to evaluate handling of volatile data.","marker":"[40]"}],"fun_headline_variants":["Act-Now framework slashes streaming forecast error by 28%","Leakage-free online learning lifts forecast accuracy 28%","Streaming forecasting gets a 28% boost without peeking ahead","New Act-Now model updates on the fly, cuts error 28%","Large-scale stream forecasting: 28% better with Act-Now"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole fast-update loop assumes that a prediction made from recent data is more accurate than one made from older data; if recent predictions are biased or noisy—say, right after a sudden change—the model learns from its own mistakes and the gains vanish.","fun_headline_variants_meta":{"raw":{"variants":["Act-Now framework slashes streaming forecast error by 28%","Leakage-free online learning lifts forecast accuracy 28%","Streaming forecasting gets a 28% boost without peeking ahead","New Act-Now model updates on the fly, cuts error 28%","Large-scale stream forecasting: 28% better with Act-Now"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1366,"prompt_tokens":1025,"completion_tokens":341,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":249}},"tokens_in":641,"tokens_out":341,"duration_ms":3965,"temperature":1.0,"reasoning_tokens":249,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:50:12.088588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a stream with sudden, repeated regime shifts, and compare Act-Now with FSB enabled versus disabled: if MSE and pseudo-label error rise together when drift is abrupt, Proposition 1 fails. More directly, at each FSB update, check whether the newer prediction's error is actually lower than the older prediction's on the eventual ground truth; if it is not, the pseudo-label target is misleading.","supporting_citations":[{"cited_title":"Learning fast and slow for online time series forecasting,","cited_arxiv_id":null,"evidence_quote":"Defines the FSNet baseline whose fast and slow learning is compared and is said to become ineffective once information leakage is removed."},{"cited_title":"Self-improving reactive agents based on reinforcement learn- ing, planning and teaching,","cited_arxiv_id":null,"evidence_quote":"Provides the Experience Replay baseline that Act-Now is compared against for cumulative MSE on the large-scale datasets."},{"cited_title":"Online continual learning with maximal interfered retrieval,","cited_arxiv_id":null,"evidence_quote":"Defines MIR, the continual-learning baseline that Lade's average 19.5% MSE reduction on Milano is measured against."},{"cited_title":"Dark experience for general continual learning: a strong, simple baseline,","cited_arxiv_id":null,"evidence_quote":"Defines DER++, a dark experience replay baseline used in the online forecasting comparison."},{"cited_title":"Online convex programming and generalized infinitesi- mal gradient ascent,","cited_arxiv_id":null,"evidence_quote":"Provides the OnlineTCN baseline based on online convex programming."},{"cited_title":"The problem of concept drift: definitions and related work,","cited_arxiv_id":null,"evidence_quote":"Defines concept drift, the problem the Lade model and the online update scheme are designed to handle."}],"review_version":1}