{"id":"51322890-b149-4353-abb5-5b5a30cc15f7","arxiv_id":"2509.03240","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A window-based F1 metric that rewards detections within a time tolerance reveals statistically significant stress-event prediction where pointwise F1 scores are all zero.","lead":"The paper proposes a new scoring rule for stress-detection algorithms from wearables, giving partial credit when a predicted stress event lands near, but not exactly on, the labeled moment. Its empirical study shows this \"windowed\" score can find statistically significant predictive skill where standard F1 scores are all zero.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proposed F1_w counts each predicted point inside an event window as a separate true positive (Table 1), so a model that emits a dense prediction block around every annotated event can score F1_w = 1 without identifying distinct events; the claimed significance over baselines may be an artifact…","rationale":"The reader's weakest assumption matches my own. The paper's central empirical claim — that only F1_w reveals statistically significant improvements on ADARP and Wrist Angel — is established through the metrics defined in Table 1, and the TP formula's point-counting semantics is the least secure condition for that claim. If an event-level counting rule changes the conclusion, the metric's practical recommendation to use F1_w to reveal meaningful model performance is not supported as stated; the metric instead rewards alert density. The paper has independent strengths: the metric is clearly specified, the synthetic illustrations are helpful, the code and predictions are released, and the authors disclose both the label-leakage limitation and the absence of point-precision adjustment. Those strengths do not resolve the counting concern, because the Discussion's caveat is not quantified. The proper response is the same CONDITIONAL verdict the reader reached: the central claim should not be accepted as a general evaluation recommendation until the TP rule is compared with an event-level rule and the significance tests are rerun. I therefore leave the verdict unchanged.","tokens_in":14147,"tokens_out":5782,"duration_ms":58732,"concrete_test":"Recompute Table 4 using an event-level F1_w variant with the same window tolerance but a one-to-one matching rule: sort true events by time; for each true event in order, match it to the nearest unused predicted positive point within w_t; count each matched pair as one TP, each unmatched predicted point as one FP, and each unmatched true event as one FN. Re-run the subject-level permutation tests for ADARP and Wrist Angel across w in {10s, 30s, 5min, 20min} using the released predictions and labels. The concern is confirmed if the significance of TimesFM over the random baseline disappears or the F1_w advantage drops to the level of the random baseline; it is refuted if the event-level F1_w remains significant at alpha = 0.05.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 defines TP_w = Σ_{t:ŷ_t=1} 1(∃t′∈w_t: y_t′=1) and FN_w = Σ_{t:y_t=1} 1(∀t′∈w_t: ŷ_t′=0). Because TP_w is summed over predicted positives rather than over true events, a single true event can generate arbitrarily many true positives: any model that emits a continuous block of positive predictions covering the window around every true event has TP_w equal to the block size, FP_w=0, FN_w=0, and hence F1_w=1 for any window size. The metric therefore rewards prediction density near events, not event detection. On ADARP and Wrist Angel, the F1_w scores grow monotonically with w and are the basis for the paper's headline significance claim; without an event-level counting rule (e.g., each true event can be matched to at most one predicted point, and each predicted point used at most once), the reported TimesFM-versus-baseline improvements may simply reflect that TimesFM emits more positive points inside event windows than the random or null baselines do. The Discussion acknowledges that 'we have not incorporated a point-precision adjustment into F1_w' and suggests a stricter precision window as future work, but the paper does not test whether the central claim survives such an adjustment. This is the load-bearing soft spot: the metric's counting semantics, not the statistical machinery, determines whether the claim that only F1_w reveals significant improvements is true.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes F1_w, a window-based F1 metric for evaluating event detection in time series where ground-truth events are annotated as single points. The metric credits a predicted point as a true positive if it falls within a window of a true event, and counts a true event as missed only if no prediction falls in its window. The authors evaluate F1_w alongside standard and point-adjusted F1 variants using zero-shot TimesFM predictions on three physiological datasets (ADARP, ROAD, Wrist Angel). They report that on the two in-the-wild datasets, only F1_w yields statistically significant improvements of TimesFM over random and null baselines, while standard metrics are all zero. The paper also discusses the role of window size as a domain-interpretable temporal tolerance and acknowledges limitations concerning the model setup and the lack of a point-precision adjustment.","tokens_in":14415,"tokens_out":4545,"duration_ms":43195,"significance":"The idea of a temporally tolerant F1 metric for point-annotated event detection is practically motivated and would be useful if its counting semantics were sound. The paper is clearly written, provides code, uses real wearable datasets, and gives a fairly thorough comparison to existing metrics. However, the central empirical claim—that F1_w uniquely reveals significant model performance—rests on a true-positive definition that rewards prediction density near events rather than event detection. Because this flaw is structural and directly affects the headline significance results, the contribution is not yet established; a corrected event-level counting scheme and re-evaluation would be needed.","major_comments":[{"comment":"The definition TP_w = Σ_{t:ŷ_t=1} 1(∃t'∈w_t: y_t'=1) counts every predicted positive point inside a window of a true event as a separate true positive. Consequently, a model that emits a continuous block of positive predictions covering the window around each true event has TP_w equal to the total block size, with FP_w=0 and FN_w=0, yielding F1_w=1 irrespective of whether the model identifies the correct number of distinct events. F1_w therefore rewards prediction density, not event detection. Since the abstract and Section 5.2 claim that only F1_w reveals statistically significant TimesFM improvements on ADARP and Wrist Angel, the claim may be an artifact of this counting rule. The Discussion (Section 6) acknowledges that no point-precision adjustment was incorporated, but the paper does not test an alternative event-level counting rule; the authors should add such a rule (e.g., each true event can be matched to at most one predicted point, and each predicted point used at most once) and report whether the significance results survive.","section":"Section 3.2, Table 1"},{"comment":"The permutation tests compare per-subject F1_w values between TimesFM and baselines, but because F1_w is inflated by the number of positive predictions in event windows, the tests effectively compare prediction density near events. This concern is amplified by the modeling setup: the stress labels themselves are used as the TimesFM target signal, and Section 6 admits this 'can result in artificially high probabilities around annotated events.' A baseline that emits the same volume of positive predictions but distributed uniformly will score lower F1_w even if neither model actually detects distinct events. The authors should report event-level precision/recall (or a capped TP_w) for both TimesFM and the baselines and recompute the permutation tests under that counting scheme; without this, the central significance claim is not established.","section":"Section 5.2, Table 4; Section 3.3; Appendix A"}],"minor_comments":[{"comment":"The null baseline is described only as 'set to yield an F-score of 0,' but the actual construction is not specified. The permutation test against this baseline is not fully reproducible without the exact definition.","section":"Section 3.4"},{"comment":"The metric is asymmetric: TP_w is summed over predicted points, while FN_w is summed over true event points. The paper should justify this asymmetry or align the two summations, since it contributes directly to the inflation described in the major comments.","section":"Table 1"},{"comment":"The hatched bars for the random baseline are difficult to see for near-zero values on ADARP and Wrist Angel; adding numerical annotations or a zoomed inset would improve readability.","section":"Figure 2"},{"comment":"The text says the standard F1 for the random baseline is 'around 0.5,' but with event prevalence of 57% in ROAD, a random classifier would be expected to score near the prevalence; please clarify whether this value is empirical or theoretical.","section":"Section 5.1, ROAD bullet"}],"recommendation":"major_revision","confidential_remarks":"The structural flaw in the TP_w definition is serious and directly affects the abstract's headline claim. However, the paper's core idea—temporal tolerance in evaluation—is salvageable, and the authors have already collected the data and code needed to test an event-level counting variant. I recommend major revision rather than rejection, provided the authors re-run the analysis with a corrected counting rule and either confirm or retract the significance claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The contribution is a window-based F1 variant that credits predictions within w time steps of a true event, plus a systematic comparison across three stress datasets using TimesFM against random and null baselines. The authors define the metric precisely in Table 1, release code, use public data for two of three datasets, and include anonymized labels and predictions for Wrist Angel. They are also upfront that feeding past stress labels as input to TimesFM is unrealistic for deployment. That transparency is real and earns credit.\n\nThe soft spot is the counting semantics, and it is load-bearing. TP_w sums over predicted positive time points, so one true event can generate arbitrarily many true positives. A model that emits a solid block of positive predictions covering every event window gets F1_w = 1 with zero event-level discrimination, because FP_w and FN_w can both be zero. The paper acknowledges the lack of point-precision adjustment, but it never tests an event-level matching rule, e.g., each true event matched to at most one prediction and each prediction used once. Without that, the significant TimesFM-versus-baseline gaps on ADARP and Wrist Angel may simply reflect that TimesFM emits more positive points inside event windows than the random baseline does, not that it detects more distinct events. The monotonic growth of F1_w with w on those datasets reinforces the suspicion. This is not a minor quibble; it determines whether the headline claim is true.\n\nOther concerns are minor by comparison. Testing four window sizes without multiple-testing correction should be addressed, but the permutation testing itself is reasonable. Threshold selection on validation is acceptable. The ROAD result is handled honestly, including the degeneracy at large w.\n\nWho is this for? Researchers evaluating point-annotated event detection in wearables, especially in mental-health monitoring. The paper raises a legitimate problem and provides a clear, reproducible baseline metric definition. But the metric as defined should not be adopted as standard until the counting semantics are fixed and the significance claims are re-run under event-level matching. I would send it to peer review; a serious referee can push for those changes. As is, I would not cite it as a recommended evaluation metric, though I would cite it as an example of why counting rules matter.","headline":"F1_w is a clearly defined temporal-tolerance metric and the paper is transparent, but its TP counting rule rewards prediction density around events, and the headline significance claims rest on that rule.","tokens_in":15002,"tokens_out":1873,"would_cite":false,"duration_ms":20060,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes F1w, a window-based F1 metric that credits predictions near a true event, and reports that on two in-the-wild stress datasets it is the only metric that separates TimesFM from random and null baselines.","keywords":["time series event detection","stress monitoring","window-based F1 metric","temporal tolerance","wearable sensors","foundation models for time series","point-adjusted F1","evaluation metrics"],"falsifier":"Recompute F1w on ADARP and Wrist Angel with an event-level true-positive rule: for each true event, count at most one success, for example the first predicted point inside the window, recompute precision, recall, and F1w, and rerun the subject-level permutation tests at 10s and 30s windows. If the significant separation from random and null baselines disappears, the paper's headline result rests on counting multiple predictions of the same event as multiple successes.","tokens_in":13922,"feed_emoji":"⏱️","tokens_out":10687,"duration_ms":87817,"temperature":0.7,"pith_summary":"Stress events in wearable data are usually annotated as single time points even though the underlying physiological response unfolds over time, so exact-match scoring punishes any model that detects the event slightly early or late. This paper argues that standard F1 and point-adjusted F1 can therefore report zero skill when real skill exists, and introduces F1w, a window-based F1 metric that counts a prediction as correct if it falls within w time steps of a true event. Using zero-shot forecasts from TimesFM on two self-annotated wearable datasets, ADARP and Wrist Angel, the authors find that all conventional metrics score zero while F1w rises with the window and is statistically significant against both random and null baselines. On the ROAD driving dataset the same metric saturates because long dense event segments make even the random baseline score high, so the paper cautions that window size and event prevalence must be checked together. The point is that evaluation choices, not just models, determine whether a stress detector appears to work.","feed_headline":"Window-based F1 score uncovers hidden stress-detection skill","feed_subtitle":"Standard F1 scores are zero on both self-annotated datasets; only the windowed metric beats random and null baselines.","key_machinery":"The central machinery is the window-based F1 definition in Table 1: true positives are counted per predicted point, not per true event, with a predicted point credited if a true label exists within its window w_t; F1w is then the standard harmonic mean of precision and recall under that counting. The window size w is the free parameter, interpretable directly as the accepted temporal lag between prediction and annotation. The statistical machinery is a subject-level permutation test with sign-flipped differences against random and null baselines, where significance is reported only if the model beats both, and bootstrap confidence intervals for the mean difference.","core_discovery":"The central claim is that event detection in point-annotated time series should be scored with temporal tolerance, and that an F1 metric built on that tolerance can reveal performance that exact and point-adjusted metrics hide. F1w defines a true positive as a predicted event point with at least one true label inside its window w; a false positive as a predicted point with no true label in the window; and a false negative as a true point with no predicted point in its window. On the two in-the-wild datasets, this definition turns zero scores into meaningful scores: ADARP F1w goes from 0.010 at a 10-second window to 0.798 at 20 minutes, and Wrist Angel from 0.056 to 0.459, with subject-level permutation tests significant at all four tested windows. On ROAD, where 57% of test points are labeled events, F1w and the random baseline both approach 1, so the paper reports that high prevalence makes the metric non-discriminative and that adjusted metrics must be interpreted against a random baseline. The authors also note that using past stress labels as the forecast target can inflate probabilities around annotated events, though the metric comparison itself is consistent across models.","pith_inferences":["Beyond the paper: the per-point true-positive rule means several predictions inside the same event window all count as successes; an event-level variant that credits at most one detection per true event would reveal whether the reported significance survives that stricter counting.","Beyond the paper: the window size could be chosen automatically as the smallest w at which the model-vs-random permutation test turns significant, which would reduce the arbitrariness of selecting 10s, 30s, 5min, or 20min.","Beyond the paper: the same windowed scoring should transfer to other point-annotated health events, such as seizures, falls, or panic episodes, where annotations mark a moment but the underlying event has duration.","Beyond the paper: the authors' own limitation (Section 6) is that TimesFM receives past stress labels as its target signal, so part of the nonzero F1w may come from short-term label autocorrelation rather than physiology-based detection; this does not weaken the metric-comparison conclusion, but it does mean the absolute performance numbers should not be read as a deployable detector's skill."],"forward_implications":["On self-annotated in-the-wild stress datasets, standard F1 and F1pa can report zero even when a model's predictions cluster near true events; F1w provides a graded, nonzero reading of that skill.","The size of the window changes the conclusion: with w=10s ADARP shows a small significant gap, while w=5min and w=20min show large gaps and near-perfect precision, so window choice must be grounded in domain knowledge.","On dense-label datasets such as ROAD, tolerance-based metrics can make even a random baseline look accurate, so claims of good performance must be paired with a random-baseline comparison.","F1w evaluates predictions directly against the original point annotations, so it removes the need to expand ground-truth labels into segments before assessment, which should improve reproducibility across studies."],"supporting_citations":[{"why":"Defines point adjustment (F1pa), the exact-match evaluation method whose zero scores on ADARP and Wrist Angel motivate F1w.","marker":"Xu et al. [2018]"},{"why":"Defines F1pa%K, the partial-credit adjusted metric that F1w is compared against and that recovers pointwise F1 at K=1.","marker":"Kim et al. [2022]"},{"why":"Introduces detection probability and detection lag, the earlier temporal-tolerance ideas that F1w turns into a precision-recall score.","marker":"Lima et al. [2024]"},{"why":"Provides the ADARP dataset and its self-annotated point-event stress labels used in the central comparison.","marker":"Sah et al. [2022]"},{"why":"Provides the Wrist Angel dataset, the second in-the-wild dataset where conventional metrics score zero and F1w finds significance.","marker":"Lønfeldt [2023]"},{"why":"Provides the ROAD driving dataset, whose dense long events show when adjusted metrics overestimate.","marker":"Haouij et al. [2018]"},{"why":"Supplies the TimesFM zero-shot forecaster whose predictions are scored by F1w across all three datasets.","marker":"Das et al. [2024]"}],"fun_headline_variants":["Windowed F1 turns zero stress scores into real signal","F1 with time window reveals stress model beats chance","Time-tolerant F1 exposes hidden stress detection skill","Temporal tolerance in F1 metric rescues stress detection scores","Time-buffer F1 metric surfaces stress event detection gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every predicted point inside the window of a true event is an independent successful detection, so a model that emits many predictions around one event accumulates many true positives; if that counting is not the right way to credit detections, the reported F1w significance could be an artifact of the counting rule.","fun_headline_variants_meta":{"raw":{"variants":["Windowed F1 turns zero stress scores into real signal","F1 with time window reveals stress model beats chance","Time-tolerant F1 exposes hidden stress detection skill","Temporal tolerance in F1 metric rescues stress detection scores","Time-buffer F1 metric surfaces stress event detection gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000679,"raw_usage":{"total_tokens":3121,"prompt_tokens":1014,"completion_tokens":2107,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":2028}},"tokens_in":630,"tokens_out":2107,"duration_ms":14355,"temperature":1.0,"reasoning_tokens":2028,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:32:10.998857+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute F1w on ADARP and Wrist Angel with an event-level true-positive rule: for each true event, count at most one success, for example the first predicted point inside the window, recompute precision, recall, and F1w, and rerun the subject-level permutation tests at 10s and 30s windows. If the significant separation from random and null baselines disappears, the paper's headline result rests on counting multiple predictions of the same event as multiple successes.","supporting_citations":[{"cited_title":"Online event detection in streaming time series: Novel metrics and practical insights","cited_arxiv_id":null,"evidence_quote":"Introduces detection probability and detection lag, the earlier temporal-tolerance ideas that F1w turns into a precision-recall score."}],"review_version":2}