REVIEW 2 major objections 4 minor 26 references
Evaluation of Stress Detection as Time Series Events -- A Novel Window-Based F1-Metric
T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper proposes F1w, a window-based F1 metric that credits predictions near a true event, and reports that on two in-the-wild stress datasets it is the only metric that separates TimesFM from random and null baselines.
desk verdict F1_w is a clearly defined temporal-tolerance metric and the paper is transparent, but its TP counting rule rewards prediction density around events, and the headline significance claims rest on that rule. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the window-based F1 definition in Table 1: true positives are counted per predicted point, not per true event, with a predicted point credited if a true label exists within its window w_t; F1w is then the standard harmonic mean of precision and recall under that counting. The window size w is the free parameter, interpretable directly as the accepted temporal lag between prediction and annotation. The statistical machinery is a subject-level permutation test with sign-flipped differences against random and null baselines, where significance is reported only if the model beats both, and bootstrap confidence intervals for the mean difference.
What would settle it
Recompute F1w on ADARP and Wrist Angel with an event-level true-positive rule: for each true event, count at most one success, for example the first predicted point inside the window, recompute precision, recall, and F1w, and rerun the subject-level permutation tests at 10s and 30s windows. If the significant separation from random and null baselines disappears, the paper's headline result rests on counting multiple predictions of the same event as multiple successes.
Extended reading notes
Core claim
The central claim is that event detection in point-annotated time series should be scored with temporal tolerance, and that an F1 metric built on that tolerance can reveal performance that exact and point-adjusted metrics hide. F1w defines a true positive as a predicted event point with at least one true label inside its window w; a false positive as a predicted point with no true label in the window; and a false negative as a true point with no predicted point in its window. On the two in-the-wild datasets, this definition turns zero scores into meaningful scores: ADARP F1w goes from 0.010 at a 10-second window to 0.798 at 20 minutes, and Wrist Angel from 0.056 to 0.459, with subject-level permutation tests significant at all four tested windows. On ROAD, where 57% of test points are labeled events, F1w and the random baseline both approach 1, so the paper reports that high prevalence makes the metric non-discriminative and that adjusted metrics must be interpreted against a random baseline. The authors also note that using past stress labels as the forecast target can inflate probabilities around annotated events, though the metric comparison itself is consistent across models.
Load-bearing premise
The load-bearing premise is that every predicted point inside the window of a true event is an independent successful detection, so a model that emits many predictions around one event accumulates many true positives; if that counting is not the right way to credit detections, the reported F1w significance could be an artifact of the counting rule.
Editorial extensions
If this is right
- On self-annotated in-the-wild stress datasets, standard F1 and F1pa can report zero even when a model's predictions cluster near true events; F1w provides a graded, nonzero reading of that skill.
- The size of the window changes the conclusion: with w=10s ADARP shows a small significant gap, while w=5min and w=20min show large gaps and near-perfect precision, so window choice must be grounded in domain knowledge.
- On dense-label datasets such as ROAD, tolerance-based metrics can make even a random baseline look accurate, so claims of good performance must be paired with a random-baseline comparison.
- F1w evaluates predictions directly against the original point annotations, so it removes the need to expand ground-truth labels into segments before assessment, which should improve reproducibility across studies.
Reading between the lines
- Beyond the paper: the per-point true-positive rule means several predictions inside the same event window all count as successes; an event-level variant that credits at most one detection per true event would reveal whether the reported significance survives that stricter counting.
- Beyond the paper: the window size could be chosen automatically as the smallest w at which the model-vs-random permutation test turns significant, which would reduce the arbitrariness of selecting 10s, 30s, 5min, or 20min.
- Beyond the paper: the same windowed scoring should transfer to other point-annotated health events, such as seizures, falls, or panic episodes, where annotations mark a moment but the underlying event has duration.
- Beyond the paper: the authors' own limitation (Section 6) is that TimesFM receives past stress labels as its target signal, so part of the nonzero F1w may come from short-term label autocorrelation rather than physiology-based detection; this does not weaken the metric-comparison conclusion, but it does mean the absolute performance numbers should not be read as a deployable detector's skill.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes F1_w, a window-based F1 metric for evaluating event detection in time series where ground-truth events are annotated as single points. The metric credits a predicted point as a true positive if it falls within a window of a true event, and counts a true event as missed only if no prediction falls in its window. The authors evaluate F1_w alongside standard and point-adjusted F1 variants using zero-shot TimesFM predictions on three physiological datasets (ADARP, ROAD, Wrist Angel). They report that on the two in-the-wild datasets, only F1_w yields statistically significant improvements of TimesFM over random and null baselines, while standard metrics are all zero. The paper also discusses the role of window size as a domain-interpretable temporal tolerance and acknowledges limitations concerning the model setup and the lack of a point-precision adjustment.
Significance. The idea of a temporally tolerant F1 metric for point-annotated event detection is practically motivated and would be useful if its counting semantics were sound. The paper is clearly written, provides code, uses real wearable datasets, and gives a fairly thorough comparison to existing metrics. However, the central empirical claim—that F1_w uniquely reveals significant model performance—rests on a true-positive definition that rewards prediction density near events rather than event detection. Because this flaw is structural and directly affects the headline significance results, the contribution is not yet established; a corrected event-level counting scheme and re-evaluation would be needed.
major comments (2)
- [Section 3.2, Table 1] The definition TP_w = Σ_{t:ŷ_t=1} 1(∃t'∈w_t: y_t'=1) counts every predicted positive point inside a window of a true event as a separate true positive. Consequently, a model that emits a continuous block of positive predictions covering the window around each true event has TP_w equal to the total block size, with FP_w=0 and FN_w=0, yielding F1_w=1 irrespective of whether the model identifies the correct number of distinct events. F1_w therefore rewards prediction density, not event detection. Since the abstract and Section 5.2 claim that only F1_w reveals statistically significant TimesFM improvements on ADARP and Wrist Angel, the claim may be an artifact of this counting rule. The Discussion (Section 6) acknowledges that no point-precision adjustment was incorporated, but the paper does not test an alternative event-level counting rule; the authors should add such a rule (e.g., each true event can be matched to at most one predicted point, and each predicted point used at most once) and report whether the significance results survive.
- [Section 5.2, Table 4; Section 3.3; Appendix A] The permutation tests compare per-subject F1_w values between TimesFM and baselines, but because F1_w is inflated by the number of positive predictions in event windows, the tests effectively compare prediction density near events. This concern is amplified by the modeling setup: the stress labels themselves are used as the TimesFM target signal, and Section 6 admits this 'can result in artificially high probabilities around annotated events.' A baseline that emits the same volume of positive predictions but distributed uniformly will score lower F1_w even if neither model actually detects distinct events. The authors should report event-level precision/recall (or a capped TP_w) for both TimesFM and the baselines and recompute the permutation tests under that counting scheme; without this, the central significance claim is not established.
minor comments (4)
- [Section 3.4] The null baseline is described only as 'set to yield an F-score of 0,' but the actual construction is not specified. The permutation test against this baseline is not fully reproducible without the exact definition.
- [Table 1] The metric is asymmetric: TP_w is summed over predicted points, while FN_w is summed over true event points. The paper should justify this asymmetry or align the two summations, since it contributes directly to the inflation described in the major comments.
- [Figure 2] The hatched bars for the random baseline are difficult to see for near-zero values on ADARP and Wrist Angel; adding numerical annotations or a zoomed inset would improve readability.
- [Section 5.1, ROAD bullet] The text says the standard F1 for the random baseline is 'around 0.5,' but with event prevalence of 57% in ROAD, a random classifier would be expected to score near the prevalence; please clarify whether this value is empirical or theoretical.
Circularity Check
No circular derivation: F1_w is introduced as a new evaluation definition, the headline claim is an empirical comparison under that metric, and the cited self-work is not load-bearing.
full rationale
The proposed F1_w metric is defined in Table 1 as a new evaluation rule and is not derived from, or fitted to, the data used in the empirical claim. The paper's central assertion that 'only our temporally tolerant metrics reveal statistically significant improvements' is an empirical statement about TimesFM predictions when scored with that definition; it is not a quantity that is equal by construction to the metric's inputs. Model thresholds are selected on validation data (Section 3.3, Table 2) and applied to held-out test splits (Section 4.2.1), so there is no fitted-parameter-renamed-as-prediction. Self-citations (Lønfeldt et al., 2023; Skat-Rørdam et al., 2024; Das et al., 2024) are used for dataset description and preprocessing choices, not as the justification for the metric or for the significance tests, and thus are not load-bearing. The Discussion explicitly acknowledges the main weaknesses: 'we have not incorporated a point-precision adjustment into F1_w' and 'Overly broad windows can overestimate performance'; these are validity and selection-effect limitations, not circular inputs. The counting semantics of TP_w (summed over predicted positives rather than matched events) can reward dense prediction blocks, but this is a design weakness that could be addressed by an event-level matching rule and does not make the empirical evaluation equivalent to its own definitions. The derivation is therefore self-contained.
Assumptions & free parameters
free parameters (4)
- Threshold delta for ADARP =
0.501
- Threshold delta for ROAD =
0.71
- Threshold delta for Wrist Angel =
0.501
- Window size w =
10s, 30s, 1min, 5min, 20min, 60min
assumptions (3)
- standard math Standard precision, recall, and F1 definitions are valid for binary event prediction.
- domain assumption Point annotations are the ground truth for stress events, and temporal proximity to a point within window w indicates correct detection.
- domain assumption TimesFM with a sigmoid output, using past stress labels as the forecast target, provides a meaningful testbed for comparing evaluation metrics.
Cite this review
Pith. "Pith review of Evaluation of Stress Detection as Time Series Events -- A Novel Window-Based F1-Metric." pith.science (2026). https://pith.science/paper/FBGRXXBP
@misc{pith2026250903240,
author = {Pith},
title = {Pith review of: Evaluation of Stress Detection as Time Series Events -- A Novel Window-Based F1-Metric},
year = {2026},
howpublished = {\url{https://pith.science/paper/FBGRXXBP}},
note = {Machine review of arXiv:2509.03240}
}
abstract
Accurate evaluation of event detection in time series is essential for applications such as stress monitoring with wearable devices, where ground truth is typically annotated as single-point events, even though the underlying phenomena are gradual and temporally diffused. Standard metrics like F1 and point-adjusted F1 (F1$_{pa}$) often misrepresent model performance in such real-world, imbalanced datasets. We introduce a window-based F1 metric (F1$_w$) that incorporates temporal tolerance, enabling a more robust assessment of event detection when exact alignment is unrealistic. Empirical analysis in three physiological datasets, two in-the-wild (ADARP, Wrist Angel) and one experimental (ROAD), indicates that F1$_w$ reveals meaningful model performance patterns invisible to conventional metrics, while its window size can be adapted to domain knowledge to avoid overestimation. We show that the choice of evaluation metric strongly influences the interpretation of model performance: using predictions from TimesFM, only our temporally tolerant metrics reveal statistically significant improvements over random and null baselines in the two in-the-wild use cases. This work addresses key gaps in time series evaluation and provides practical guidance for healthcare applications where requirements for temporal precision vary by context.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[3]
Note that the stress signal lags behind the covariates. TimesFM incorporates covariates that are available at prediction time, aligning them with the forecast horizon. By shifting the stress signal forward by the length of the horizon window, the model uses covariates to predict future stress values, rather than relying on information about the current st...
work page 2025
-
[6]
Context = 2048 (512 seconds, 8.5 minutes) horizon = 256 (64 seconds)
0 3 6 9 1e6 0.480 0.485 0.490 0.495 0.500 0.505 0.510 0.515 0.520 Non-event Event =0.501 Figure 4: ADARP predicted probabilities. Context = 2048 (512 seconds, 8.5 minutes) horizon = 256 (64 seconds). 13 A novel window-based F1-metric A PREPRINT 0 10000 20000 30000 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 Non-event Event =0.71 Figure 5: ROAD predicted proba...
work page 2007
-
[7]
URL https://arxiv.org/abs/2109.05257. David Hand and Peter Christen. A note on using the f-measure for evaluating record linkage algorithms. Statistics and Computing, 28(3):539–547,
-
[8]
URL https://doi.org/10.1007/s11222 -017-9746-6
doi:10.1007/s11222-017-9746-6. URL https://doi.org/10.1007/s11222 -017-9746-6 . Rob J. Hyndman and George Athanasopoulos. F orecasting: Principles and Practice. OTexts, 2 edition,
-
[10]
doi:10.14778/3538598.3538602. Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. Large language models are few-shot health learners,
-
[11]
Wei-Bang Jiang, Li-Ming Zhao, and Bao-Liang Lu
URL https://arxiv.org/abs/2305.15525. Wei-Bang Jiang, Li-Ming Zhao, and Bao-Liang Lu. Large brain model for learning generic representations with tremendous eeg data in bci,
-
[12]
Jiexia Ye, Weiqi Zhang, Ke Yi, Yongzi Yu, Ziyue Li, Jia Li, and Fugee Tsung
URL https://arxiv.org/abs/2405.18765. Jiexia Ye, Weiqi Zhang, Ke Yi, Yongzi Yu, Ziyue Li, Jia Li, and Fugee Tsung. A survey of time series foundation models: Generalizing time series representation with large language model,
-
[13]
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou
URL https://arxiv.org/ab s/2405.02358. Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting,
Show all 26 references
-
[14]
Valery Guralnik and Jaideep Srivastava
URL https://arxiv.org/abs/2310.10688. Valery Guralnik and Jaideep Srivastava. Event detection from time series data. In Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD ’99, page 33–42, New York, NY , USA,
-
[16]
doi:10.1007/s10044-017-0657-0
ISSN 1433-7541. doi:10.1007/s10044-017-0657-0. URL https://doi.org/10.1007/s10044-017-0657-0 . Nancy Chinchor and Beth Sundheim. MUC-5 Evaluation Metrics. In Fifth Message Understanding Conference (MUC-5): Proceedings of a Conference Held in Baltimore, Maryland, August 25-27, 1993 ,
-
[17]
doi:10.1109/tnnls.2021.3105827
ISSN 2162-2388. doi:10.1109/tnnls.2021.3105827. URL http://dx.doi.org/10. 1109/TNNLS.2021.3105827. Medical Device Coordination Group. MDCG 2021-24 guidance on classification of medical devices,
2021
-
[18]
Empatica
URL https://health.ec.europa.eu/system/files/2021-10/mdcg_2021-24_en_0.pdf . Empatica. Empatica legal & compliance,
2021
-
[19]
Accessed: 02-01-2024
URL https://www.empatica.com/legal. Accessed: 02-01-2024. Neska El Haouij, Jean-Michel Poggi, Sylvie Sevestre-Ghalila, Raja Ghozi, and Mériem Jaïdane. Affectiveroad system and database to assess driver’s attention. In Proceedings of the 33rd Annual ACM Symposium on Applied Com...
2024
-
[20]
ISBN 9781450351911
Association for Computing Machinery. ISBN 9781450351911. doi:10.1145/3167132.3167395. URL https://doi.org/10.1145/3167132.3167395. Cristina Bustos, Neska El Haouij, Albert Solé-Ribalta, Javier Borge-Holthoefer, Àgata Lapedriza, and Rosalind W. Picard. Predicting driver self-re...
-
[21]
Pagsberg A Lønfeldt, N; Clemmensen L
URL https://arxiv.org/abs/2109.13225. Pagsberg A Lønfeldt, N; Clemmensen L. A wearable artificial intelligence feedback tool (wrist angel) for treatment and research of obsessive compulsive disorder: Protocol for a nonrandomized pilot study. JMIR Res Protoc, 12,
-
[22]
URL https://www.researchprotocols.org/2023/1/e45123
doi:10.2196/45123. URL https://www.researchprotocols.org/2023/1/e45123. Simon Haykin and Barry Van Veen. Signals and systems. John Wiley & Sons,
2023 doi
-
[23]
Harald Vilhelm Skat-Rørdam, Mia Hang Knudsen, Simon Nørby Knudsen, Sneha Das, and Line H Clemmensen
URL https://www.ucsfbenioffchildrens.org/medical-tes ts/pulse. Harald Vilhelm Skat-Rørdam, Mia Hang Knudsen, Simon Nørby Knudsen, Sneha Das, and Line H Clemmensen. Data augmentations and transfer learning for physiological time series. In ICLR 2024 Workshop on Learning from Ti...
2024
-
[1999]
ISBN 1581131437
Association for Computing Machinery. ISBN 1581131437. doi:10.1145/312129.312190. URL https://doi.org/10.1145/312129.312190. André Gensler and Bernhard Sick. Performing event detection in time series with swiftevent: an algorithm with supervised learning of detection criteria. ...
-
[2004]
Online event detection in streaming time series: Novel metrics and practical insights
Janio Lima, Lucas Giusti Tavares, Esther Pacitti, João Eduardo Ferreira, Ismael Santos, Isabela Guimarães Siqueira, Diego Carvalho, Fabio Porto, Rafaelli Coutinho, and Eduardo Ogasawara. Online event detection in streaming time series: Novel metrics and practical insights. In ...
2024
-
[2018]
URL http://arxiv.org/abs/1802.03903
doi:10.1145/3178876.3185996. URL http://arxiv.org/abs/1802.03903. arXiv:1802.03903 [cs]. Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, and Sungroh Yoon. Towards a rigorous evaluation of time-series anomaly detection,
-
[2019]
URL https://pubmed.ncbi.nlm.nih
doi:10.3390/s19194079. URL https://pubmed.ncbi.nlm.nih. gov/31547220/. Sally S Dickerson and Margaret E Kemeny. Acute stressors and cortisol responses: a theoretical integration and synthesis of laboratory research. Psychological bulletin, 130(3):355,
-
[2021]
Nature Scientific Data
URL https://doi.org/10.1038/s41597-022-01361-y . Nature Scientific Data. Rahul Gupta, Kartik Audhkhasi, Sungbok Lee, and Shrikanth S. Narayanan. Paralinguistic event detection from speech using probabilistic time-series smoothing and masking. In Interspeech,
-
[2022]
Nicole Nadine
URL https: //arxiv.org/abs/2206.14568. Nicole Nadine. Lønfeldt, Kristoffer V . Olesen, Sneha Das, Anna-Rose Cecilie Mora-Jensen, Anne Ka- trine Pagsberg, and Line Katrine Harder Clemmensen (2023). Predicting obsessive compulsive disorder episodes in adolescents using wearable ...
2023 arXiv
-
[2023]
URL https://www.frontiersin.org/journals/psych iatry/articles/10.3389/fpsyt.2023.1231024/full
doi:https://doi.org/10.3389/fpsyt.2023.1231024. URL https://www.frontiersin.org/journals/psych iatry/articles/10.3389/fpsyt.2023.1231024/full. 14:1231024. Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, Jie Che...
2023
-
[2024]
Ramesh Kumar Sah, Michael McDonell, Patricia Pendry, Sara Parent, Hassan Ghasemzadeh, and Michael J Cleveland
doi:10.1109/IJCNN60899.2024.10650809. Ramesh Kumar Sah, Michael McDonell, Patricia Pendry, Sara Parent, Hassan Ghasemzadeh, and Michael J Cleveland. Adarp: A multi modal dataset for stress and alcohol relapse quantification in real life setting,
2024
-
[2025]
The TimesFM architecture illustration in the yellow box is from Das et al
11 A novel window-based F1-metric A PREPRINT Predictions Input data Context length = 2048 Context length = 2048 TimesFM Fit Horizon length = 256 Physiological signals Tags Linear model on covariates Figure 3: Illustration of the modeling setup. The TimesFM architecture illustr...
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.