{"id":"b6b65105-1396-4ff9-8085-0ec1d1e0b399","arxiv_id":"2607.15423","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A simulation-calibrated detection threshold controls the probability of at least one false change-point alarm within any moving window of length Δ.","lead":"This paper defines a sequential family-wise error rate (sFWER) for online change point detection and uses simulations to pick a threshold that controls false alarms within any moving window. In simulations the calibrated threshold beats Bonferroni and Šidák corrections, and the method is illustrated on smartphone mobility data from adolescents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Later-window sFWER control is untested: cutoff is calibrated only from the first window after run-in, and Section 3.1 does not check windows starting after r+Δ.","rationale":"The reader's weakest assumption is stationarity of the monitoring statistic distribution across windows, and that is exactly the load-bearing condition I identify. The paper's Section 3.1 only checks sensitivity to run-in length, not to window start time; even within the paper's own i.i.d. Gaussian simulation model, the distribution of S_t may change as t grows because the candidate p-values are computed with increasing amounts of pre-candidate data. The paper explicitly states in the Discussion that the calibration assumes stability under the null, so the limitation is acknowledged, but it is not tested. This does not invalidate the method; it means the central claim is conditional on an unverified property. The proposed concrete test—simulating long null trajectories and checking sFWER for later windows—would directly settle whether the concern lands. Since the reader already returned CONDITIONAL and my concern reinforces that rather than overturning it, no change to the verdict is warranted.","tokens_in":9018,"tokens_out":6541,"duration_ms":69689,"concrete_test":"Run the §3.2 null simulation with total length T >> r+Δ (e.g., r=30, Δ=7, T=100). Calibrate c from the first window exactly as in §2.2 (B=1000 null datasets of length r+Δ). Then compute S_t for all t up to T and estimate P(min_{t+1≤j≤t+Δ} S_j ≤ c) for each start t = r+1, r+Δ+1, r+2Δ+1, ... . Plot these estimated sFWERs against t. If any later-window estimate exceeds α_Δ by more than roughly 2 standard errors (≈2√(α(1-α)/1000)), the first-window calibration does not deliver the 'any t≥r' control claimed in Eq. (1).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central guarantee in Eq. (1) is that P(∪_{j=t+1}^{t+Δ}{S_j≤c}) ≤ α_Δ for every t≥r. But the cutoff c in Section 2.2 is the empirical α_Δ quantile of M = min_{r+1≤j≤r+Δ} S_j from null simulations of length r+Δ. By construction, this controls the first monitoring window only; control for any later window requires that the joint null distribution of (S_{t+1},...,S_{t+Δ}) is the same as that of (S_{r+1},...,S_{r+Δ}). Section 3.1 varies the run-in length r, which shifts the data before the first window, but it never simulates data past t=r+Δ and never estimates the sFWER for windows that begin later. This gap is not merely about external non-stationarity. Even for i.i.d. Gaussian null data, each S_t is a minimum of permutation p-values over candidate days k∈{t-db,...,t-1}, and the null distribution of each candidate p-value depends on the amount of pre-candidate data, which grows with t. The Discussion acknowledges that the procedure 'assumes that the data are stable under the null hypothesis,' but the simulation study never verifies the specific stationarity condition that Eq. (1) requires. Until later-window false-alarm rates are measured, the claim that the calibrated cutoff controls sFWER for any moving window is supported only for the first window.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a sequential family-wise error rate (sFWER), defined as the probability of at least one false positive within a moving monitoring window of length Δ, and proposes a simulation-based calibration of a cutoff c for an online change point detection statistic. The cutoff is the empirical α_Δ quantile of the minimum of the monitoring statistic over the first monitoring window under a null model. The authors claim this controls the sFWER for any window after the run-in period, and present simulations comparing the proposed procedure with unadjusted, Bonferroni, and Šidák thresholds under the null and alternative hypotheses. They also illustrate the procedure on smartphone mobility data from an mHealth study. The central claim is that the calibrated cutoff balances error control and power better than standard corrections, which are either conservative or liberal.","tokens_in":9384,"tokens_out":3602,"duration_ms":41466,"significance":"If the central claim holds, the paper offers a practical, simulation-based alternative to Bonferroni/Šidák for online monitoring with strongly dependent test statistics. The sFWER definition is a useful formalization for mHealth and similar settings, and the simulation study is designed in a non-circular way: the cutoff is calibrated under an explicit null model and then evaluated on independent null datasets. The paper also transparently acknowledges the stationarity assumption and the illustrative nature of the real-data analysis. However, the simulations verify error control only for the first monitoring window, so the paper's principal guarantee—control for every moving window—is not empirically supported as stated.","major_comments":[{"comment":"The cutoff c is calibrated as the empirical α_Δ quantile of M = min_{r+1≤j≤r+Δ} S_j from null datasets of length r+Δ. This directly controls the first monitoring window only. Eq. (1), however, requires P(∪_{j=t+1}^{t+Δ}{S_j≤c}) ≤ α_Δ for every t≥r. Section 3.1 varies the run-in length r, which shifts the pre-window data, but never evaluates windows beginning at later times. Section 3.2's null evaluation also uses datasets of length r+Δ and checks only the window (r+1,...,r+Δ). The claim that the procedure controls the sFWER for any moving window is therefore supported only for the first window. Please add simulations with long null series (T >> r+Δ) and report the empirical false-alarm rate for windows starting at t = r+1, r+2, ..., T-Δ, or at minimum at t = r+1 and t = r+Δ+1. Without such evidence, Eq. (1) remains an assumption rather than a demonstrated property.","section":"Section 2.2 and Eq. (1)"},{"comment":"Even under i.i.d. Gaussian null data, the distribution of S_t = min_{t-db≤k≤t-1} p_k may depend on t because each candidate p-value p_k is computed using data accumulated up to time t, and the amount of pre-candidate data changes as k increases. The Discussion acknowledges that the procedure assumes the data are stable under the null, but this stationarity of the raw data does not automatically imply stationarity of the joint distribution of (S_{t+1},...,S_{t+Δ}) for different t. The paper should either provide a formal argument or empirical evidence that the null distribution of S_t is invariant across monitoring days for the VC* statistic. In particular, Section 3.1's check across run-in lengths is not a check of later-window behavior. If the null distribution of S_t changes with t, the calibrated cutoff may fail to control the sFWER for later windows even under a perfectly stationary","section":"Section 2.3 and Eq. (4)"}],"minor_comments":[{"comment":"The unadjusted procedure's rejection rates are identical for α_Δ=0.1 and α_Δ=0.2 (e.g., 0.18 for p=1, Δ=7), which is expected because it uses a fixed 0.05 threshold. This duplication is not explained in the table or text; a note would avoid confusion.","section":"Table 1"},{"comment":"The Monte Carlo sample size is B=1000. For a target α_Δ=0.1, the standard error of the estimated type I error is approximately 0.0095; for α_Δ=0.2, approximately 0.013. The observed values (e.g., 0.23 for p=10, α_Δ=0.2, Δ=7) are within reasonable Monte Carlo variability, but adding standard errors or confidence intervals to Table 1 would help the reader assess deviations from the target.","section":"Section 3.2"},{"comment":"The real-data calibration uses a pooled correlation matrix estimated from all available data, including data after potential change points. Because the analysis is retrospective and explicitly illustrative, this is not fatal, but the text should state more clearly that the null model for cutoff calibration is not independent of the data being monitored. Supplement S2 is cited for insensitivity to correlation structure, but the main text would benefit from one sentence explaining this limitation.","section":"Section 4.2"},{"comment":"The caption says error bars show variability across bootstrap samples, but the number of bootstrap resamples and the resampling scheme are not specified in the main text. Please provide those details.","section":"Figure 1 caption"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The sFWER over a moving window is a sensible operational target for online monitoring, and the calibration recipe is simple enough that practitioners could actually use it. In the simulations, it lands near the nominal rate and beats both the unadjusted threshold (too liberal) and Bonferroni/Šidák (too conservative). That part is real and clearly presented.\n\nThe main gap is the one the stress test flags: Eq. (1) promises control for every t≥r, but the cutoff is calibrated from the first window after run-in and never evaluated on windows that begin later. Section 3.1 varies run-in length only; it does not simulate data beyond r+Δ or estimate false-alarm rates for subsequent windows. The Discussion does admit that the procedure assumes the data are stable under the null, but that is precisely the assumption that needs testing. Even for i.i.d. Gaussian null data, the distribution of S_t could shift as t grows because each candidate p-value depends on how much pre-candidate data exists. So this is not an irrelevant edge case; it is a load-bearing assumption left unverified. It is fixable with additional simulations, but as written the central claim is only supported for the first window.\n\nSmaller issues: no code or data are provided, so the calibration procedure is not directly reproducible. The real-data illustration is retrospective, uses full-data correlation estimates for the cutoff, and is honestly presented as an illustration rather than validation. That is fine, but it should not be oversold.\n\nNone of this is fatal. The sFWER definition and the empirical-quantile calibration are new enough, and the simulation evidence is solid for the tested settings. The paper is coherent and honest about several limitations, which makes me think the authors are aware of the boundaries of what they have shown. The missing later-window analysis is the one thing an editor should insist on before publication. If it holds up in longer null simulations, this could become a standard reference for false-alarm control in mHealth monitoring.\n\nMy recommendation: send it to peer review. The right referee will push on the stationarity issue, but the paper deserves the push.","headline":"A genuinely useful new error-rate definition and a simple simulation-based calibration that works in the settings tested, but the paper never checks the central claim that control holds for every later monitoring window, not just the first.","tokens_in":9859,"tokens_out":1620,"would_cite":true,"duration_ms":21163,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62L10","62F03","62P10"],"pacs":[],"model":"deepseek-v4-flash","headline":"A simulation-quantile threshold can control false alarms in online change point detection over any moving monitoring window.","keywords":["online change point detection","sequential family-wise error rate","multiple testing","monitoring threshold calibration","null simulation","moving window","mobile health","variance component score test"],"falsifier":"Simulate a no-change series with slow drift or autocorrelation, calibrate c under the paper's i.i.d. null model, then record the fraction of later Δ-windows (starting at various t beyond r) that contain at least one rejection; if that fraction clearly exceeds α_Δ, the sFWER guarantee fails.","tokens_in":8899,"feed_emoji":"🎯","tokens_out":6576,"duration_ms":57317,"temperature":0.7,"pith_summary":"Online change point detection performs a new hypothesis test on each arriving observation, so false alarms accumulate in a way standard multiple-testing corrections do not handle. This paper introduces the sequential family-wise error rate (sFWER), the probability of at least one false alarm within any moving window of length Δ, and proposes to calibrate a single monitoring cutoff from simulated null data. The cutoff is the empirical α_Δ quantile of the distribution of the minimum of the monitoring statistics over the window. Simulation studies show the cutoff holds the sFWER near its target while common alternatives are either too conservative (fixed per-test corrections) or too liberal (unadjusted day-by-day thresholds). If correct, practitioners can choose a detection threshold directly from a clinically meaningful false-alarm window.","feed_headline":"Null-quantile cutoff controls false alarms in change point tests","feed_subtitle":"Standard multiplicity corrections are too strict; unadjusted testing is too lax. A simulated quantile lands on target.","key_machinery":"The key object is the window-minimum statistic M, the smallest of the Δ daily monitoring statistics in a sliding window. Because the daily statistics use overlapping data, their joint distribution—and hence the distribution of M—has no closed form. The paper approximates this distribution by simulation under the null and defines the cutoff c as the empirical α_Δ quantile of M; a change is declared on day t whenever S_t ≤ c. In the demonstrations, S_t is the VC* monitoring statistic, defined as the minimum permutation p-value over candidate change-point days in the last d_b days.","core_discovery":"The paper's central claim is that under the global null, the distribution of M = min_{t+1≤j≤t+Δ} S_j — the smallest daily monitoring statistic in a window — determines a correct monitoring cutoff. Generate B null datasets of length r+Δ, compute M for each, and set c to the empirical α_Δ quantile of these values. Then for every t ≥ r, the probability that the window {t+1,...,t+Δ} contains a false alarm is approximately α_Δ. The paper demonstrates the claim for the VC* statistic, a variance-component score test whose monitoring statistic is the minimum permutation p-value over recent candidate change-point days, and reports that the proposed threshold hits the target while corrections that ign","pith_inferences":["The sFWER guarantee is only as good as the stationarity of the null: the cutoff is calibrated from the first window, and the paper's sensitivity check varies run-in length but never monitors later windows under a drifting null. Testing that directly would settle how robust the threshold is.","Because only the window-minimum distribution is used, the same calibration recipe should transfer to other change-point statistics (CUSUM, likelihood-ratio, etc.) with no theoretical change—only the simulation step differs.","For special null models (e.g., Gaussian AR processes), an analytic approximation of the window-minimum distribution might replace the simulation, removing Monte Carlo uncertainty and making the method lighter for high-frequency monitoring.","The real-data application resets monitoring after each detected change; the stated sFWER bound applies within one reset segment rather than across the whole follow-up, so cumulative false-alarm risk over an entire study may be higher than α_Δ."],"forward_implications":["Practitioners can set a monitoring threshold directly from a desired false-alarm rate per Δ-day window, without knowing the number of future tests.","The same cutoff is claimed to control the sFWER for every window after the run-in, not just the calibration window, provided the null distribution is stationary.","The calibrated threshold preserves substantially more detection power than corrections that ignore the dependence among overlapping test statistics.","The procedure applies to multivariate data and can be adapted to any test statistic whose null window-minimum distribution can be simulated.","In daily mHealth monitoring the extra computation is small enough to be practical, since only null simulations are needed up front."],"fun_headline_variants":["Simulated null quantile tames false alarms","Windowed error control for online change points","Calibrated thresholds fix real-time false positives","Null-distribution cutoff hits error target in streaming","Sequential FWER controlled via simulation calibration"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The distribution of the monitoring statistic under 'no change' is the same for every window after the run-in, so a threshold learned from the first window keeps its error rate later.","fun_headline_variants_meta":{"raw":{"variants":["Simulated null quantile tames false alarms","Windowed error control for online change points","Calibrated thresholds fix real-time false positives","Null-distribution cutoff hits error target in streaming","Sequential FWER controlled via simulation calibration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00012,"raw_usage":{"total_tokens":906,"prompt_tokens":705,"completion_tokens":201,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":142}},"tokens_in":449,"tokens_out":201,"duration_ms":2913,"temperature":1.0,"reasoning_tokens":142,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T23:23:37.079711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate a no-change series with slow drift or autocorrelation, calibrate c under the paper's i.i.d. null model, then record the fraction of later Δ-windows (starting at various t beyond r) that contain at least one rejection; if that fraction clearly exceeds α_Δ, the sFWER guarantee fails.","supporting_citations":[],"review_version":1}