pith. sign in

arxiv: 2605.20621 · v1 · pith:RDST64K2new · submitted 2026-05-20 · 📊 stat.ME · stat.AP· stat.CO

Changepoint Detection in Categorical Time Series with Application to Daily Total Cloud Cover in Canada

Mo Li , QiQi Lu , XiaoLan Wang This is my paper

Pith reviewed 2026-05-21 03:13 UTC · model grok-4.3

classification 📊 stat.ME stat.APstat.CO
keywords changepoint detectioncategorical time seriesmarginalized transition modelMarkov chainlikelihood ratio testcloud cover dataperiodic seriesserial correlation
0
0 comments X

The pith

A marginalized transition model detects single changepoints in periodic categorical time series by modeling serial dependence with a first-order Markov chain.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents a statistical approach for finding abrupt shifts in the proportions of different categories within time series that exhibit daily or seasonal patterns and dependence from one observation to the next. Traditional methods often aggregate data to yearly levels to reduce these issues, but that loses detail and can create overdispersion. The new method keeps the daily observations by using a model where the probability of each category can jump at a changepoint, while transitions between days follow a simple Markov process. A custom estimation method finds the best-fitting parameters efficiently, and a special test statistic checks whether a change is present and where it occurs. When applied to daily cloud cover records from Canada, this allows analysis of trends in clear, partly cloudy, and overcast conditions without simplifying the series too much.

Core claim

The authors introduce a marginalized transition model that specifies category-specific marginal probabilities which may include a single changepoint, combined with a first-order Markov chain to account for serial correlation in periodic categorical time series. They provide a new procedure to obtain maximum likelihood estimates and propose a maximally selected likelihood ratio test for detecting the presence of a sudden change. The approach is illustrated with daily total cloud cover observations at 9 a.m. and 3 p.m. from Fort St. John Airport in British Columbia.

What carries the argument

The marginalized transition model, which allows the marginal distribution of each category to shift at a specified changepoint while using a first-order Markov chain to capture the dependence between successive observations in the series.

If this is right

  • The model preserves the full daily resolution of the time series instead of requiring annual aggregation to handle seasonality and correlation.
  • Changepoints can be specified separately for each category of the response variable.
  • The estimation procedure reduces computational burden for obtaining the maximum likelihood estimates.
  • The maximally selected likelihood ratio test provides a formal way to assess evidence for a sudden change in the categorical frequencies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the first-order Markov assumption holds, the method could extend to other environmental or health categorical series with periodic patterns.
  • Future work might adapt the framework to detect multiple changepoints or incorporate higher-order dependence if needed.
  • Applying the test to historical data could help identify climate-related shifts in cloud cover patterns at specific locations.

Load-bearing premise

The serial dependence in the time series is fully captured by a first-order Markov chain, and marginalization suffices to address periodicity and overdispersion without needing more complex dependence structures or multiple changepoints.

What would settle it

Generating synthetic categorical time series from the model both with and without a planted changepoint, then checking whether the maximally selected likelihood ratio test correctly identifies the change location and avoids false positives when none exists.

read the original abstract

Changepoints are essential for homogenizing categorical time series and analyzing their trends and variations. The original total cloud cover in Canada was recorded hourly in tenths (or eighths), exhibiting inherent seasonality and serial correlation. Lu and Wang (2012) introduced an extended cumulative logit model to detect shifts in the annual frequencies of cloud cover conditions. While annual aggregation mitigates seasonality and serial correlation, it shortens the time series and may lead to overdispersion. This article introduces a marginalized transition model to detect a single changepoint in periodic and serially correlated categorical time series. The model captures serial dependence using a first-order Markov chain and enables category-specific changepoint specification. To enhance computational efficiency, we develop a new parameter estimation procedure for obtaining maximum likelihood estimates. A maximally selected likelihood ratio test statistic is then proposed to test for sudden changes in categorical time series, and the method is illustrated using daily total cloud cover observations recorded at 9 a.m. and 3 p.m. at Fort St. John Airport, British Columbia, Canada.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript introduces a marginalized transition model for single changepoint detection in periodic categorical time series that exhibit serial correlation. Serial dependence is captured via a first-order Markov chain, with category-specific changepoint specification permitted. A computationally efficient procedure is developed for maximum likelihood estimation, followed by a maximally selected likelihood ratio test for detecting abrupt changes. The method is illustrated on daily total cloud cover observations recorded at 9 a.m. and 3 p.m. at Fort St. John Airport, British Columbia.

Significance. If the modeling assumptions hold, the approach improves upon annual aggregation methods by retaining daily resolution while addressing seasonality and serial dependence through marginalization. The new MLE procedure and the maximally selected LR test constitute practical methodological contributions for categorical time series. The application to Canadian meteorological data supplies a concrete, real-world demonstration of utility in homogenizing cloud cover records.

major comments (1)
  1. [Model specification and assumptions (Section 2)] The central modeling claim—that marginalization of the first-order Markov transition fully absorbs periodicity and overdispersion while preserving a correctly specified conditional distribution—underpins both the MLE procedure and the validity of the maximally selected LR test. In daily cloud cover series, unmodeled multi-day persistence or residual periodicity after marginalization would bias the likelihood ratio and invalidate its null distribution; the manuscript provides no diagnostic checks or robustness analysis against higher-order dependence.
minor comments (2)
  1. [Abstract and data description] The abstract states that the original data were recorded in tenths or eighths but does not specify how the categories are coded or reduced for the transition model; this detail should be added for reproducibility.
  2. [Notation and model equations] Notation for the marginalized transition probabilities and the changepoint parameter could be introduced earlier and used consistently to improve readability of the estimation and test sections.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback and positive evaluation of the methodological contributions and real-world application. We address the single major comment below and will revise the manuscript accordingly to strengthen the presentation of model assumptions and validation.

read point-by-point responses

  1. Referee: [Model specification and assumptions (Section 2)] The central modeling claim—that marginalization of the first-order Markov transition fully absorbs periodicity and overdispersion while preserving a correctly specified conditional distribution—underpins both the MLE procedure and the validity of the maximally selected LR test. In daily cloud cover series, unmodeled multi-day persistence or residual periodicity after marginalization would bias the likelihood ratio and invalidate its null distribution; the manuscript provides no diagnostic checks or robustness analysis against higher-order dependence.

    Authors: We agree that the validity of the MLE and the null distribution of the maximally selected LR test rests on the modeling assumptions. The marginalized transition model is constructed so that the marginal distribution (which incorporates the category-specific changepoint and periodic structure) is correctly specified while the serial dependence is captured by a first-order Markov chain; this is a standard device in the categorical time-series literature to accommodate overdispersion without inflating the parameter count. Under the null of no changepoint the likelihood ratio therefore has the expected asymptotic behavior conditional on the assumed dependence structure. Nevertheless, the referee correctly notes that the manuscript currently lacks explicit diagnostics for residual higher-order dependence or multi-day persistence. In the revision we will add (i) a simulation study comparing the test’s size and power under first-order versus second-order Markov data-generating processes and (ii) residual autocorrelation plots and a formal comparison with a higher-order alternative on the Fort St. John data. These additions will be placed in a new subsection of Section 4. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation is self-contained

full rationale

The paper develops a new marginalized transition model that incorporates a first-order Markov chain for serial dependence in periodic categorical time series, along with a custom MLE procedure and a maximally selected likelihood ratio test for single changepoint detection. These elements are constructed directly from standard Markov transition probabilities and likelihood theory without reducing to fitted inputs or prior results by definition. The citation to Lu and Wang (2012) is used solely for motivating the drawbacks of annual aggregation (seasonality, overdispersion) and does not supply any load-bearing uniqueness theorem, ansatz, or parameter that the new model depends upon. No self-definitional loops, renamed empirical patterns, or self-citation chains appear in the core derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on standard time series assumptions for categorical data; no free parameters or invented entities are identifiable from the abstract alone.

axioms (2)
  • domain assumption Serial dependence in the categorical time series is captured by a first-order Markov chain.
    Invoked to model correlation between consecutive daily observations while handling periodicity.
  • domain assumption The series contains at most one changepoint that can be specified per category.
    Underlies the design of the detection procedure and test statistic.

pith-pipeline@v0.9.0 · 5719 in / 1425 out tokens · 64595 ms · 2026-05-21T03:13:57.399154+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

95 extracted references · 95 canonical work pages

  1. [1]

    2002 , publisher =

    Alan Agresti , title =. 2002 , publisher =. doi:10.1002/0471249688 , url =

    work page doi:10.1002/0471249688 2002

  2. [2]

    Biometrika , volume=

    Logistic regression for autocorrelated data with application to repeated measures , author=. Biometrika , volume=. 1994 , publisher=

    work page 1994

  3. [3]

    Glossary of Meteorology , year =

    Cloud cover. Glossary of Meteorology , year =

    work page

  4. [4]

    International Journal of Climatology: A Journal of the Royal Meteorological Society , volume=

    HISTALP—historical instrumental climatological surface time series of the Greater Alpine Region , author=. International Journal of Climatology: A Journal of the Royal Meteorological Society , volume=. 2007 , publisher=

    work page 2007

  5. [5]

    GeoJournal , volume=

    Continental cloudiness changes this century , author=. GeoJournal , volume=. 1992 , publisher=

    work page 1992

  6. [6]

    Climate of the Past , volume=

    Increasing cloud cover in the 20th century: review and new findings in Spain , author=. Climate of the Past , volume=. 2012 , publisher=

    work page 2012

  7. [7]

    The annals of statistics , pages=

    Some asymptotic theory for the bootstrap , author=. The annals of statistics , pages=. 1981 , publisher=

    work page 1981

  8. [8]

    Neural Computing and Applications , volume=

    Machine learning for total cloud cover prediction , author=. Neural Computing and Applications , volume=. 2021 , publisher=

    work page 2021

  9. [9]

    Kybernetika , volume=

    On Bartlett's test for correlation between time series , author=. Kybernetika , volume=. 1998 , publisher=

    work page 1998

  10. [10]

    Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , volume=

    Change-point analysis as a tool to detect abrupt climate variations , author=. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , volume=. 2012 , publisher=

    work page 2012

  11. [11]

    1953 , url =

    Environment and Climate Change Canada , title =. 1953 , url =

    work page 1953

  12. [12]

    2024 , month =

    Cloud Forecast for astronomical observations , author =. 2024 , month =

    work page 2024

  13. [13]

    2018 , publisher=

    Analysis of binary data , author=. 2018 , publisher=

    work page 2018

  14. [14]

    Cs. Limit. 1997 , publisher=

    work page 1997

  15. [15]

    Biometrics , volume=

    Predictive model assessment for count data , author=. Biometrics , volume=. 2009 , publisher=

    work page 2009

  16. [16]

    Computational Statistics & Data Analysis , volume=

    Dealing with overdispersion in multivariate count data , author=. Computational Statistics & Data Analysis , volume=. 2022 , publisher=

    work page 2022

  17. [17]

    Journal of Climate , volume=

    Changepoint detection in climate time series with long-term trends , author=. Journal of Climate , volume=

    work page

  18. [18]

    Environmetrics: The official journal of the International Environmetrics Society , volume=

    Some problems with application of change-point detection methods to environmental data , author=. Environmetrics: The official journal of the International Environmetrics Society , volume=. 1997 , publisher=

    work page 1997

  19. [19]

    Recent trends in cloudiness over the

    Dai, Aiguo and Karl, Thomas R and Sun, Bomin and Trenberth, Kevin E , journal=. Recent trends in cloudiness over the. 2006 , publisher=

    work page 2006

  20. [20]

    1991 , publisher=

    Handbook of genetic algorithms , author=. 1991 , publisher=

    work page 1991

  21. [21]

    Journal of the American Statistical Association , volume=

    Structural break estimation for nonstationary time series models , author=. Journal of the American Statistical Association , volume=. 2006 , publisher=

    work page 2006

  22. [22]

    SIAM review , volume=

    Computers and the theory of statistics: thinking the unthinkable , author=. SIAM review , volume=. 1979 , publisher=

    work page 1979

  23. [23]

    Multivariate

    Fahrmeir, Ludwig and Tutz, Gerhard , address =. Multivariate

    work page

  24. [24]

    Biometrika , volume=

    A likelihood-based method for analysing longitudinal binary responses , author=. Biometrika , volume=. 1993 , publisher=

    work page 1993

  25. [25]

    2012 , publisher=

    Applied longitudinal analysis , author=. 2012 , publisher=

    work page 2012

  26. [26]

    Journal of Statistical Planning and Inference , volume=

    Retrospective change detection for binary time series models , author=. Journal of Statistical Planning and Inference , volume=. 2014 , publisher=

    work page 2014

  27. [27]

    Time-varying biases in

    Free, Melissa and Sun, Bomin , journal=. Time-varying biases in. 2013 , publisher=

    work page 2013

  28. [28]

    The Annals of Statistics , volume=

    Wild binary segmentation for multiple change-point detection , author=. The Annals of Statistics , volume=. 2014 , publisher=

    work page 2014

  29. [29]

    2001 , publisher=

    Multivariate statistical modelling based on generalized linear models , author=. 2001 , publisher=

    work page 2001

  30. [30]

    Giudici, Paolo and Givens, Geof H and Mallick, Bani K , year=

    work page

  31. [31]

    Biometrika , volume=

    Asymptotic distributions of maximum likelihood tests for change in the mean , author=. Biometrika , volume=. 1990 , publisher=

    work page 1990

  32. [32]

    Communications in Statistics-Theory and Methods , volume=

    Retrospective change detection in categorical time series , author=. Communications in Statistics-Theory and Methods , volume=. 2017 , publisher=

    work page 2017

  33. [33]

    Environmetrics , volume=

    Changepoint detection in daily precipitation data , author=. Environmetrics , volume=. 2012 , publisher=

    work page 2012

  34. [34]

    Journal of the Korean Statistical Society , volume=

    Autocovariance estimation in the presence of changepoints , author=. Journal of the Korean Statistical Society , volume=. 2022 , publisher=

    work page 2022

  35. [35]

    Journal of hydrometeorology , volume=

    Contemporary changes of the hydrological cycle over the contiguous United States: Trends derived from in situ observations , author=. Journal of hydrometeorology , volume=. 2004 , publisher=

    work page 2004

  36. [36]

    J. V. Braun and R. K. Braun and H. -G. Muller , journal =. Multiple Changepoint Fitting via Quasilikelihood, with Application to

    work page

  37. [37]

    Journal of the American Statistical Association , volume=

    Latent Gaussian count time series , author=. Journal of the American Statistical Association , volume=. 2023 , publisher=

    work page 2023

  38. [38]

    Biometrics , pages=

    Maximally selected chi square statistics for small samples , author=. Biometrics , pages=. 1982 , publisher=

    work page 1982

  39. [39]

    Biometrics , volume=

    Marginalized transition models and likelihood inference for longitudinal categorical data , author=. Biometrics , volume=. 2002 , publisher=

    work page 2002

  40. [40]

    Monthly Weather Review , volume=

    Discrete postprocessing of total cloud cover ensemble forecasts , author=. Monthly Weather Review , volume=

    work page

  41. [41]

    Biometrika , volume=

    Conditional bootstrap methods in the mean-shift model , author=. Biometrika , volume=. 1987 , publisher=

    work page 1987

  42. [42]

    Scandinavian journal of statistics , pages=

    Testing for changes in multinomial observations: the Lindisfarne Scribes problem , author=. Scandinavian journal of statistics , pages=. 1995 , publisher=

    work page 1995

  43. [43]

    Journal of Climate , volume=

    Homogenization of daily temperature data , author=. Journal of Climate , volume=

    work page

  44. [44]

    Mausam , volume=

    Changes in total cloud cover over India based upon 1961-2007 surface observations , author=. Mausam , volume=

    work page 1961

  45. [45]

    1975 , publisher=

    Adaptation in natural and artificial systems , author=. 1975 , publisher=

    work page 1975

  46. [46]

    A high-quality monthly total cloud amount dataset for

    Jovanovic, Branislava and Collins, Dean and Braganza, Karl and Jakob, Doerte and Jones, David A , journal=. A high-quality monthly total cloud amount dataset for. 2011 , publisher=

    work page 2011

  47. [47]

    Biometrics , volume=

    A class of Markov models for longitudinal ordinal data , author=. Biometrics , volume=. 2007 , publisher=

    work page 2007

  48. [48]

    Computational Statistics & Data Analysis , volume=

    Longitudinal nominal data analysis using marginalized models , author=. Computational Statistics & Data Analysis , volume=. 2010 , publisher=

    work page 2010

  49. [49]

    Journal of Applied Statistics , volume=

    Likelihood-based approach for analysis of longitudinal nominal data using marginalized random effects models , author=. Journal of Applied Statistics , volume=. 2011 , publisher=

    work page 2011

  50. [50]

    Computational Statistics & Data Analysis , volume=

    Analysis of long series of longitudinal ordinal data using marginalized models , author=. Computational Statistics & Data Analysis , volume=. 2016 , publisher=

    work page 2016

  51. [51]

    Computational Statistics & Data Analysis , volume=

    Marginalized models for longitudinal count data , author=. Computational Statistics & Data Analysis , volume=. 2019 , publisher=

    work page 2019

  52. [52]

    Journal of Climate , volume=

    Multiple changepoint detection via genetic algorithms , author=. Journal of Climate , volume=. 2012 , publisher=

    work page 2012

  53. [53]

    Environmetrics , volume=

    Changepoint detection in autocorrelated ordinal categorical time series , author=. Environmetrics , volume=. 2022 , publisher=

    work page 2022

  54. [54]

    arXiv preprint arXiv:2410.15571 , year=

    changepointGA: An R package for Fast Changepoint Detection via Genetic Algorithm , author=. arXiv preprint arXiv:2410.15571 , year=

    work page arXiv

  55. [55]

    The Annals of Applied Statistics , volume=

    An MDL approach to the climate segmentation problem , author=. The Annals of Applied Statistics , volume=. 2010 , publisher=

    work page 2010

  56. [56]

    Journal of Geophysical Research: Atmospheres , volume=

    An extended cumulative logit model for detecting a shift in frequencies of sky-cloudiness conditions , author=. Journal of Geophysical Research: Atmospheres , volume=. 2012 , publisher=

    work page 2012

  57. [57]

    Wang and QiQi Lu and Jaxk Reeves and Colin Gallagher and Yang Feng

    Robert Lund and Xiaolan L. Wang and QiQi Lu and Jaxk Reeves and Colin Gallagher and Yang Feng. Changepoint Detection in Periodic and Autocorrelated Time Series. Journal of Climate. 2007

    work page 2007

  58. [58]

    Geophysical Research Letters , volume=

    Trends in Italian total cloud amount, 1951-1996 , author=. Geophysical Research Letters , volume=. 2001 , publisher=

    work page 1951

  59. [59]

    Atmosphere-Ocean , volume=

    Baseline cloudiness trends in Canada 1953--2002 , author=. Atmosphere-Ocean , volume=. 2004 , publisher=

    work page 1953

  60. [60]

    MacNeill , journal =

    Ian B. MacNeill , journal =. Tests for Change of Parameter at Unknown Times and Distributions of Some Related Functionals on Brownian Motion , volume =

    work page

  61. [61]

    Statistics in Medicine , volume=

    Marginal modelling of multivariate categorical data , author=. Statistics in Medicine , volume=. 1999 , publisher=

    work page 1999

  62. [62]

    2021 , publisher=

    Introduction to linear regression analysis , author=. 2021 , publisher=

    work page 2021

  63. [63]

    1989 , publisher=

    Generalized linear models , author=. 1989 , publisher=

    work page 1989

  64. [64]

    Circular binary segmentation for the analysis of array-based

    Olshen, Adam B and Venkatraman, ES and Lucito, Robert and Wigler, Michael , journal=. Circular binary segmentation for the analysis of array-based. 2004 , publisher=

    work page 2004

  65. [65]

    Geophysical research letters , volume=

    Increased cloudiness in the United States during the first half of the twentieth century: Fact or fiction? , author=. Geophysical research letters , volume=. 1990 , publisher=

    work page 1990

  66. [66]

    Analysis of total cloud amount over

    Kaiser, Dale P , journal=. Analysis of total cloud amount over. 1998 , publisher=

    work page 1998

  67. [67]

    Decreasing cloudiness over

    Kaiser, Dale P , journal=. Decreasing cloudiness over. 2000 , publisher=

    work page 2000

  68. [68]

    Regression

    Kedem, Benjamin and Fokianos, Konstantinos , year=. Regression

    work page

  69. [69]

    Computers & Chemical Engineering , volume=

    Bayesian and Expectation Maximization methods for multivariate change point detection , author=. Computers & Chemical Engineering , volume=. 2014 , publisher=

    work page 2014

  70. [70]

    International Journal of Forecasting , volume=

    Evaluating predictive count data distributions in retail sales forecasting , author=. International Journal of Forecasting , volume=. 2016 , publisher=

    work page 2016

  71. [71]

    Water SA , volume=

    Trends in cloud cover from 1960 to 2005 over South Africa , author=. Water SA , volume=. 2007 , publisher=

    work page 1960

  72. [72]

    Statistics and Its Interface , volume=

    A variable selection approach to multiple change-points detection with ordinal data , author=. Statistics and Its Interface , volume=. 2020 , publisher=

    work page 2020

  73. [73]

    PloS One , volume=

    Structural change detection in ordinal time series , author=. PloS One , volume=. 2021 , publisher =

    work page 2021

  74. [74]

    Biometrics , pages=

    Maximally selected chi square statistics , author=. Biometrics , pages=. 1982 , publisher=

    work page 1982

  75. [75]

    Journal of Computational and Graphical Statistics , volume=

    An autoregressive ordered probit model with application to high-frequency financial data , author=. Journal of Computational and Graphical Statistics , volume=. 2005 , publisher=

    work page 2005

  76. [76]

    Gender, race, pay and promotion in the

    Pudney, Stephen and Shields, Michael , journal=. Gender, race, pay and promotion in the. 2000 , publisher=

    work page 2000

  77. [77]

    Maximally Selected ^

    Betensky, Rebecca A and Rabinowitz, Daniel , journal=. Maximally Selected ^. 1999 , publisher=

    work page 1999

  78. [78]

    Changepoints in the

    Robbins, Michael W and Lund, Robert B and Gallagher, Colin M and Lu, QiQi , journal=. Changepoints in the. 2011 , publisher=

    work page 2011

  79. [79]

    Journal of Time Series Analysis , volume=

    Mean shift testing in correlated data , author=. Journal of Time Series Analysis , volume=. 2011b , publisher=

    work page

  80. [80]

    Journal of the American Statistical Association , volume=

    A general regression changepoint test for time series data , author=. Journal of the American Statistical Association , volume=. 2016 , publisher=

    work page 2016

Showing first 80 references.