{"id":"fd3f651d-0d8c-462d-9970-ebd5b6fedb99","arxiv_id":"2502.06584","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"This paper introduces Alert*, a deep-RL trigger for early time-series classification, and reports it significantly outperforms existing hand-designed triggering rules when delay costs are high.","lead":"Many time-series systems must decide when to stop collecting data and make a prediction. This paper trains reinforcement-learning agents to make that stop-or-wait decision, and shows that a full-feature version beats several existing early-classification rules on many benchmark datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Alert_Eco's state omits time and expected-cost inputs used by Economy, so the claimed 'same information' comparison (Sec. 6.3) is not established; re-running with Economy's actual inputs would settle whether man-tailored rules really beat RL on small state spaces.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: Alert_Eco does not receive the same information as Economy. This is the most serious issue because it directly undermines the paper's first contribution ('a thorough comparison of man-tailored vs. RL-based triggering functions') and the conclusion that 'man-tailored rules performed well overall, especially when input space remains small.' The main Alert* vs. SOTA claim is supported by Wilcoxon tests, and after applying Holm correction the alpha > 0.5 comparisons in Table 2 survive, so that central empirical result is not obviously threatened. The dataset-count inconsistency (20 + 114 vs. 31) is a reproducibility issue but likely a typo and less scientifically load-bearing. The Economy state-space mismatch, by contrast, is a concrete, checkable flaw in the comparison methodology. The proposed test would settle it: give the RL agent the same cost-expectation and time features that Economy actually uses. If the RL agent then performs differently, the paper's stated 'exactly the same information' guarantee is false. The verdict should remain conditional: the main finding may be correct, but this issue must be resolved before the paper's full claims can be accepted without qualification.","tokens_in":32781,"tokens_out":11546,"duration_ms":100382,"concrete_test":"Re-run the Economy vs. Alert_Eco comparison with Alert_Eco's state space extended to include the same inputs used by Economy's triggering rule: the full posterior probabilities, the current time t, and, if feasible, the precomputed expected cost vector. Keep all training, validation, and evaluation settings otherwise identical. If Economy still wins at similar rates for alpha >= 0.5, the paper's conclusion stands; if Alert_Eco matches or outperforms Economy, the 'same information' claim is false and the win rates in Fig. 3a must be reinterpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's first contribution is a fair comparison between man-tailored triggering rules and RL counterparts given 'exactly the same information' (Sec. 6.3). For Economy, however, the man-tailored rule triggers by comparing expected costs computed from posterior probabilities and the cost functions (Sec. 4.1), while the RL counterpart Alert_Eco is given only {level of confidence} as its state space (Fig. 3a). Moreover, the level of confidence is a binned maximum posterior and does not include the current time t, even though the delay cost in the reward depends on t. Without time in the state, the MDP is non-Markovian: the agent cannot distinguish the delay increment it is paying when it chooses 'wait'. Thus the RL agent is not receiving the same decision-relevant information as Economy, and the comparison is not 'all other things being equal.' The conclusion that man-tailored rules beat their RL counterparts on small state spaces, which motivates the larger Alert* state space, is therefore not established. This does not by itself refute the main Alert* vs. SOTA result, but it weakens a central claim of the paper and the narrative explaining why Alert* performs well.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses two questions in early classification of time series (ECTS): (1) whether reinforcement-learning-based triggering rules can outperform man-tailored decision rules when both use the same classifier and the same feature set, and (2) whether a larger, hand-selected feature set improves an RL-based trigger. The authors formulate separable ECTS as a reinforcement learning problem with a reward equal to the negative of the misclassification-plus-delay cost, instantiate RL counterparts for Stopping Rule, Economy, Calimera, and Proba Threshold, and compare them on 31 datasets. They then introduce Alert*, an RL trigger whose state space combines all features used by these baselines, and report that it has the best mean rank over the whole alpha range, with statistical significance for alpha > 0.5. The paper concludes that Alert* significantly outperforms state-of-the-art competitors for all weighted combinations of the two costs.","tokens_in":33059,"tokens_out":3325,"duration_ms":32990,"significance":"If the main empirical result holds, the paper makes a useful contribution: it provides a reusable RL formulation of separable ECTS triggering, a controlled comparison protocol with a shared classifier, and a strong new trigger, Alert*, supported by Wilcoxon tests and bootstrap confidence intervals. The code is released, which is a concrete strength. The controlled comparison for the Economy baseline, however, is not actually controlled, and the conclusion overstates the significance of the Alert* result. These issues are fixable and do not invalidate the main Alert* versus state-of-the-art comparison, but they must be addressed before the paper can be accepted.","major_comments":[{"comment":"The conclusion states that Alert* 'significantly outperformed its state-of-the-art competitors for all weighted combinations' of misclassification and delay costs, but this is contradicted by the paper's own Table 2: for alpha <= 0.5, many p-values are above 0.05, and Section 6.5.2 explicitly acknowledges that differences for alpha <= 0.5 are not significant. The conclusion should be restricted to the range alpha > 0.5 or rephrased to report the actual significance pattern over alpha.","section":"Section 7 and Appendix A.3, Table 2"},{"comment":"The claim that each RL counterpart receives 'exactly the same information' as its man-tailored baseline is not satisfied for Economy. Economy's rule compares expected costs computed from posterior probabilities and both cost functions over current and future times (Section 4.1), whereas Alert_Eco's state is only {level of confidence} (Figure 3a), a binned maximum posterior that does not include the current time t. Since the reward in Eq. (7) contains delay-cost increments that depend on t, the MDP is not Markovian without time in the state, and the RL agent cannot recover the delay increment it pays when choosing 'wait'. The resulting comparison does not isolate the decision rule, and the conclusion that man-tailored rules beat their RL counterparts when the input space is small is not established by this experiment. Re-running Alert_Eco with Economy's actual inputs, including t and the expected-cost features, would settle the question.","section":"Section 6.3 and Figure 3a"}],"minor_comments":[{"comment":"The dataset count is inconsistent: the text says '31 datasets: 20 from the UCR archive and 114 from the Monash time series extrinsic regression archive', which cannot sum to 31; this should be corrected to 11 (or the selection procedure should be clarified).","section":"Section 6.1"},{"comment":"The caption says the Wilcoxon signed-rank test is 'labeled with mean A vgCost', which is unclear; it should state whether the labels are mean ranks or mean costs, and how they relate to the displayed test.","section":"Figure 4b caption"},{"comment":"The model selection procedure is described only verbally; a pseudocode or a precise statement of the epoch grid, the number of splits, and the criterion for 'best-performing one over all splits' would improve reproducibility.","section":"Section 5.4"},{"comment":"The notation Alert* (with asterisk) is used in Section 5.3 and later, but Figure 4a and the conclusion sometimes render it as 'Alert⋆'; please unify the notation.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the main Alert* versus state-of-the-art result appears defensible. The two major issues—the overclaim in the conclusion and the non-equivalent Economy comparison—are fixable with text changes and, ideally, an additional experiment using Economy's actual state inputs. I do not see a novelty disclosure problem; the code release is a positive aspect."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper has a real finding and a real warts. The finding is that an RL-based trigger with a rich feature set (Alert*) beats the standard hand-designed triggers on the time-series early-classification benchmark at high delay-cost weights. The warts are an overstated conclusion and a flawed 'same information' comparison for one baseline.\n\nWhat's genuinely new: the authors translate early classification into a clean RL setup where the state space mirrors the features used by existing methods, then do a controlled comparison of man-tailored and learned triggers. That is a useful methodology. Alert* itself is a sensible combination of features and the experiments are extensive: 30+ datasets, multiple baselines, Wilcoxon tests with Holm correction, bootstrap confidence intervals, Pareto analysis, and code released. For alpha > 0.5 the significance is well-supported. For alpha <= 0.5 there's no meaningful difference, and the paper admits this in the body.\n\nWhere it's soft: the conclusion says 'significantly outperformed ... for all weighted combinations' but that's not what the results show. It should be 'for most weightings with heavy delay cost' or similar. That's a straightforward fix.\n\nBigger issue: the Economy/Alert_Eco comparison. The paper claims the RL counterpart receives 'exactly the same information' (Section 6.3), but Alert_Eco's state is just {level of confidence} -- a binned max posterior, with no time and no expected costs. Economy's rule uses expected costs derived from posterior probabilities and the cost functions, which depend on time. Without time in the state, the reward for waiting is not identifiable from the state; the MDP is non-Markovian. So that specific comparison is not 'all other things being equal,' and the narrative that man-tailored rules win on small state spaces is not established. This doesn't sink the main Alert* result, but it does weaken a secondary claim and needs to be addressed, either by adding the missing features to the RL state or by rephrasing the claim.\n\nAlso there's an internal inconsistency in dataset count: Section 6.1 says 20 UCR + 114 Monash, but the abstract and conclusion say 31. Probably a typo, but it should be fixed.\n\nBottom line: the central empirical claim -- that Alert* dominates at high delay costs -- is credible and reproducible. Send it to peer review, but the authors need to clean up the overclaim and fix (or properly qualify) the Economy comparison.","headline":"A mostly solid empirical paper with a genuine result for high delay costs, undercut by an overstated conclusion and a flawed 'same information' comparison for one baseline.","tokens_in":34,"tokens_out":2875,"would_cite":true,"duration_ms":54438,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Alert*, a reinforcement-learned trigger, achieves lower average misclassification-plus-delay cost than state-of-the-art early classifiers across all cost weightings on 31 datasets.","keywords":["early classification of time series","reinforcement learning","triggering function","deep Q-network","misclassification cost","delay cost","time series","offline reinforcement learning"],"falsifier":"Retrain the Economy RL counterpart with a state that includes the full vector of expected future costs computed from the posterior probabilities and the cost functions, on the same 31 datasets and cost settings, and compare against Economy. If the enriched RL agent matches or beats Economy, the claim that man-tailored rules dominate on small state spaces fails; if it still loses, the claim survives.","tokens_in":32604,"feed_emoji":"⏱️","tokens_out":5597,"duration_ms":98259,"temperature":0.7,"pith_summary":"Early classification of time series requires a trigger that decides, before the whole series is seen, whether to wait or to predict. The paper asks whether the hand-coded rules used by existing triggers are actually good, and answers by recasting the trigger as a reinforcement learning problem with the same input features. Fair comparisons on 31 datasets show that hand-coded rules remain competitive when the state space is small, but that an RL trigger fed a combination of classifier features, Alert*, achieves lower average misclassification-plus-delay cost than all tested competitors for every cost weighting, significantly so when delay costs dominate.","feed_headline":"RL-learned trigger beats hand-tuned early-classification rules","feed_subtitle":"A deep Q-network fed classifier confidence, margin, and time cuts misclassification-plus-delay cost across 31 datasets.","key_machinery":"The central machinery is the decomposition of a triggering function into a criterion c and a decision rule r, together with the replacement of r by a learned policy. The RL formulation defines states as the classifier's feature set F, actions as wait or trigger, and rewards as the negative incremental delay cost, plus the negative misclassification cost when the trigger fires. A Double Deep Q-Network (DDQN) approximates the optimal Q-function, and its argmax policy decides when to stop. Alert* is the variant whose state space combines five features: predicted class, maximum posterior, margin, confidence-level bin, and elapsed time.","core_discovery":"The paper's central claim is that a triggering function can be learned instead of designed, and that the learned version can beat the man-tailored rules. Using a Double Deep Q-Network trained offline with rewards equal to the negative incremental delay cost and, on triggering, the misclassification cost, the authors build Alert*, whose state space combines the predicted class, the maximum posterior probability, the margin, a binned confidence level, and the elapsed time. In experiments over 31 public datasets, Alert* dominates all state-of-the-art methods on average for all values of the trade-off parameter alpha, with statistically significant wins for alpha greater than 0.5, while the hand-tailored Economy rule beats its RL counterpart when both use a very small state space.","pith_inferences":["The Economy comparison may be unfair: its RL counterpart's state is only a binned confidence level, while the Economy rule uses predicted cost expectations computed from posterior probabilities and the cost functions; giving the RL agent those expectations could erase or reverse the apparent advantage of the hand-coded rule.","The success of Alert* suggests a general recipe: for any separable ECTS method, define the state as the classifier's full output distribution plus time, train offline, and obtain a trigger that is at least as good as the hand-designed one; this could be tested on multivariate series or with non-additive costs.","The paper's finding that a conservative trigger (waiting past the a posteriori best time) can be profitable is a testable hypothesis about robustness to label noise: on cleaner datasets the optimal policy should approach the a posteriori best trigger time more closely."],"forward_implications":["Any separable early-classification system can upgrade its trigger by training a DDQN on the features its classifier already outputs, without changing the classifier.","The fair-comparison protocol gives the field a way to test whether a proposed hand-designed trigger actually improves on a learned policy given the same information.","Larger, well-chosen state spaces improve learned triggers; adding raw time-series values or random noise does not, so feature choice matters more than raw data.","At low delay costs every method triggers immediately and differences vanish, so gains from learned triggers concentrate in regimes where waiting has a real price.","End-to-end RL methods like Earliest remain competitive with strong baselines despite seeing cost only through a hyperparameter, suggesting they deserve further study."],"supporting_citations":[{"why":"Supplies the taxonomy and benchmark that define which competitors are state of the art and how they are compared.","marker":"[36]"},{"why":"Source of the Stopping Rule baseline and its max-posterior-plus-margin feature set, which Alert_SR mirrors.","marker":"[29]"},{"why":"Source of the Economy baseline, the non-myopic cost-based decision rule whose RL counterpart is Alert_Eco.","marker":"[1]"},{"why":"Source of the Calimera baseline and the regressor-based trigger whose features Alert_Cal uses.","marker":"[4]"},{"why":"Source for the predicted-class feature that is included in Alert*'s state space.","marker":"[41]"},{"why":"The end-to-end RL method Earliest used as an additional competitor despite not sharing the common classifier.","marker":"[14]"},{"why":"Supplies the common time-series classifier used by all compared triggering functions.","marker":"[9]"},{"why":"Supplies 20 of the 31 public benchmark datasets used in the experiments.","marker":"[7]"},{"why":"Supplies 11 of the 31 public benchmark datasets used in the experiments.","marker":"[43]"}],"fun_headline_variants":["RL learns better early-classification triggers than human-designed rules","Deep RL beats hand-tuned triggers for early time-series classification","Alert: RL agent masters early classification timing on 31 datasets","Learned trigger functions beat manual rules in early time-series prediction","Double Deep Q-Network redefines early classification triggers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on each RL agent seeing exactly the same information as its man-tailored counterpart, yet for Economy the RL state contains only a binned confidence level while the Economy rule computes predicted cost expectations from posterior probabilities and the cost functions; if that information gap is real, the conclusion that hand-coded rules win on small state spaces is not established.","fun_headline_variants_meta":{"raw":{"variants":["RL learns better early-classification triggers than human-designed rules","Deep RL beats hand-tuned triggers for early time-series classification","Alert: RL agent masters early classification timing on 31 datasets","Learned trigger functions beat manual rules in early time-series prediction","Double Deep Q-Network redefines early classification triggers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00056,"raw_usage":{"total_tokens":2647,"prompt_tokens":915,"completion_tokens":1732,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1647}},"tokens_in":531,"tokens_out":1732,"duration_ms":12999,"temperature":1.0,"reasoning_tokens":1647,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:56:48.737297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the Economy RL counterpart with a state that includes the full vector of expected future costs computed from the posterior probabilities and the cost functions, on the same 31 datasets and cost settings, and compare against Economy. If the enriched RL agent matches or beats Economy, the claim that man-tailored rules dominate on small state spaces fails; if it still loses, the claim survives.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the taxonomy and benchmark that define which competitors are state of the art and how they are compared."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Stopping Rule baseline and its max-posterior-plus-margin feature set, which Alert_SR mirrors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Economy baseline, the non-myopic cost-based decision rule whose RL counterpart is Alert_Eco."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Calimera baseline and the regressor-based trigger whose features Alert_Cal uses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source for the predicted-class feature that is included in Alert*'s state space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The end-to-end RL method Earliest used as an additional competitor despite not sharing the common classifier."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies 11 of the 31 public benchmark datasets used in the experiments."}],"review_version":1}