REVIEW 3 major objections 6 minor 28 references
PCA-DDReach: Efficient Statistical Reachability Analysis of Stochastic Dynamical Systems via Principal Component Analysis
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read PCA-aligned error boxes and per-segment surrogate models make statistically guaranteed reachable sets smaller, distribution-shift-safe, and scalable to thousands of steps.
desk verdict Solid extension of the conformal reachability line with a real scalability payoff; the unverified TV bound in the shift experiment is the main obstacle. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the PCA-calibrated residual $\rho = \max_j(|r_j|/\omega_j)$. The eigenvectors $V$ and the scales $\omega_j$ are computed from the training trajectories' prediction errors; projecting onto $V$ rotates the coordinate system so the error cloud is axis-aligned, and dividing by $\omega_j$ normalizes each axis. Robust conformal inference on a separate calibration set returns the scalar $\rho^*_{\delta,\tau}$, and the identity $\rho < \rho^*_{\delta,\tau} \Leftrightarrow \bigwedge_j (|r_j| < \rho^*_{\delta,\tau} \omega_j)$ turns that one-dimensional quantile into a multidimensional parallelotope. Concatenating the per-segment parallelotopes and Minkowski-adding them to the surrogate flowpipe produces the $\delta$-confident flowpipe; the same equivalence is what lets the paper state Proposition 5 as a finite conjunction over segments.
What would settle it
Run the powertrain experiment with process-noise covariance increased by 25% instead of 20%, keep $\tau = 0.04$ and $\delta = 0.95$, simulate many deployment trajectories, and count the fraction that fall outside the computed flowpipe; if that empirical miss rate exceeds 5%, the stated distribution-shift guarantee is violated. Equivalently, estimate $TV(J_{\mathrm{sim}},J_{\mathrm{real}})$ from samples and check whether it is actually below $\tau$.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that a $\delta$-confident flowpipe $X = \bar{X} \oplus \delta X$ remains valid when the residual is defined in PCA coordinates as $\rho = \max_j(|r_j|/\omega_j)$, where $r = V^\top(PE - \bar{PE})$ projects the prediction errors onto principal axes and $\omega_j$ are the maximum projected magnitudes seen in training. With a separate calibration dataset and robust conformal inference, the order statistic $\rho^*_{\delta,\tau}$ at rank $\ell^* = \lceil (L+1)(1+1/L)(\delta+\tau) \rceil$ bounds the $\delta$-quantile of the real residual distribution as long as the total variation between simulated and real residuals is below $\tau$. Proposition 5 converts that scalar bound into a conjunction of interval constraints on the projected errors, defining an inflating parallelotope $\delta X = \langle \bar{PE}, V, P \rangle$; Lemma 3 then gives the flowpipe $X = \bar{X} \oplus \delta X$. The paper reports that this yields a valid 99.99% flowpipe in the $\tau=0$ cases and a valid 95% flowpipe for the powertrain under a 20% noise-covariance increase, with visibly smaller volume than the axis-aligned hypercube baseline.
Load-bearing premise
The load-bearing premise is that the total variation between the simulator's residual distribution and the real system's residual distribution is no larger than a pre-chosen number $\tau$; in the shifted experiment $\tau = 0.04$ is stated without being estimated or verified, and if the true total variation exceeds $\tau$ the robust conformal bound in Eq. (4) no longer applies.
Editorial extensions
If this is right
- For a fixed $\delta$ and shift bound $\tau$, the PCA-oriented inflating parallelotope is contained in or smaller than the axis-aligned origin-centered hypercube of the baseline method, so the resulting safety certificates cover less volume for the same confidence.
- The per-segment training strategy removes the need to iterate a single large model over the time horizon, eliminates the associated cumulative error, and enables exact-star reachability on the small ReLU networks, scaling the approach to horizons of 5000 steps.
- The distribution-shift guarantee transfers from the simulator to deployment whenever $TV(J_{\mathrm{sim}}, J_{\mathrm{real}}) \leq \tau$, because robust conformal inference inflates the rank by $\tau$ before selecting the quantile; the paper demonstrates this on the 27-dimensional powertrain with $\tau = 0.04$ and $\delta = 0.95$.
- The method covers high-dimensional hybrid systems: the powertrain's 4000-step trajectories are handled with 4000 independent segment models and approx-star reachability, producing a flowpipe that still meets the 95% coverage guarantee.
Reading between the lines
- A practical extension the paper leaves implicit is estimating $\tau$ from data: since the guarantee needs $TV(J_{\mathrm{sim}},J_{\mathrm{real}}) \leq \tau$, one could collect a small set of real trajectories and use a histogram or kernel estimate of total variation to pick $\tau$, rather than asserting it as in the shifted experiment.
- The max-norm residual could be replaced by an $\ell^p$ or sorted-quantile norm, which would trade a slightly larger scalar bound for a tighter shape; with strongly correlated errors this might shrink the flowpipe further while keeping the conformal argument intact.
- Because the segment models are independent, the algorithm is embarrassingly parallel; choosing segment lengths adaptively from local error variability, rather than fixing $T_q = 1$, could reduce the number of models without losing exact-star efficiency.
- A nonlinear coordinate transform, such as a normalizing flow, could replace the linear PCA projection to capture curved error distributions; the cost is that the reachable set would no longer be a parallelotope, so the star-set arithmetic would need a different representation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PCA-DDReach, a data-driven statistical reachability method for stochastic dynamical systems. It replaces the axis-aligned inflating hypercube of Hashemi et al. (2024b) with a hypercube oriented along principal components of the prediction-error vectors, and it introduces a segmented training strategy in which independent ReLU surrogate models map the initial state to each trajectory segment. The residual is defined as the maximum normalized PCA-projected error, and robust conformal inference is applied on a separate calibration dataset to obtain a rank ℓ* (Eq. (4)) that accounts for distribution shift measured by a total-variation bound τ. The resulting δ-confident flowpipe is X = ¯X ⊕ δX (Section 3.2). Experiments cover a 12-dimensional quadcopter and a 27-dimensional powertrain, including one distribution-shift case with τ=0.04.
Significance. If the claimed guarantees hold, the main contribution is meaningful: a component-wise calibrated region in PCA coordinates can reduce conservatism relative to axis-aligned hypercubes, and segmented training addresses the scalability bottleneck of a single trajectory-to-trajectory surrogate model. A clear strength is that the paper uses a proper split-conformal protocol: PCA/scaling parameters are fitted on the training set, and the conformal quantile is computed on a separate i.i.d. calibration set, so the derivation in Proposition 5 does not suffer from circularity. However, the distribution-shift certificate in Experiment 3 depends on an unverified total-variation bound, and the claimed conservatism improvement over the baseline is not quantified.
major comments (3)
- [Appendix A.3, Eq. (4)] The total-variation bound τ=0.04 is asserted without any estimation, derivation, or confidence statement. The text says that a 20% increase in the process-noise covariance makes 0.04 a valid upper bound, but the residual in Eq. (12) is a maximum of normalized PCA-projected errors, so its distribution under a perturbed noise covariance is a nontrivial function of the trained surrogate and the PCA fit. If TV(J_sim, J_real) exceeds 0.04, the robust conformal rank in Eq. (4) no longer provides Pr[ρ<ρ*]>δ, and the δ-confident flowpipe in Experiment 3 is not justified. Please either estimate τ with a statistically valid upper confidence bound, derive a rigorous bound for this specific shift, or clearly position Experiment 3 as an empirical illustration without a formal guarantee.
- [Section 4.1, Table 1] The central claim that the PCA-based inflating hypercube is less conservative than the method of Hashemi et al. (2024b) is not supported by any quantitative measure. Figure 3 is a qualitative projection, and Table 1 reports only runtimes and dataset sizes, not the volume, interval width, or any scalar conservatism metric of the reachable sets. Experiments 2 and 3 have no baseline at all. Please add quantitative comparisons of set sizes for at least Experiment 1, and report volume or other size metrics for the flowpipes in all experiments.
- [Section 4] No experiment reports the empirical coverage of the computed flowpipe on hold-out trajectories from the deployment distribution. For a statistical certificate, the paper should report the fraction of test trajectories contained in X over repeated trials; this is especially important in Experiment 3, where validity depends on the unverified τ=0.04. Without such a check, the numerical evaluation cannot confirm that the method delivers its stated δ-confidence in practice.
minor comments (6)
- [Section 2.2] There are several typographical issues, e.g., 'In other word' should be 'In other words', and 'upper-bound' should be 'upper bound' in Section 2.3.
- [Section 3.1, Eq. (8)] The notation for segmented surrogates F_q(s0; θ_q) is introduced without a formal definition of the output dimension; please clarify that F_q maps to R^{T_q n} and reconcile this with the component notation F_j in Eq. (1).
- [Table 1] The table formatting is difficult to parse, e.g., entries such as '# avg runtime| T_trn |' mix dataset sizes and runtimes; please separate these into distinct columns with clear units.
- [Section 2.3, Eq. (4)] The conditions under which Eq. (4) is valid should be stated explicitly, in particular that δ+τ<1 and that L is large enough for ℓ*≤L.
- [Experiment 2, Eq. (18)] The analytical interpolation strategy for regenerating surrogate models is described only briefly; please state whether calibration residuals are computed for each interpolated model and whether those interpolated models are also used when fitting the PCA/scaling parameters.
- [Figure 2] The caption refers to red and green hypercubes but the figure appears to be grayscale; please ensure the visual distinction is clear or adjust the caption.
Circularity Check
No significant circularity: the PCA-fitted residual is calibrated on a separate dataset, so the conformal quantile is not equivalent to a fitted value; the unverified τ=0.04 in Experiment 3 is an empirical-support gap, not a circular reduction.
full rationale
The derivation chain is self-contained. The residual in Eq. (12) is defined using PCA axes, centers, and scaling factors ω_j fitted on the training set, but the paper explicitly states that reusing the training set for conformal inference would violate CI rules, and therefore defines a separate i.i.d. calibration dataset (Definition 4). The robust conformal rank in Eq. (4) is applied only to calibration residuals, so the quantile ρ* is not a re-used training value. Proposition 5's equivalence between the residual threshold and the predicate P is definitional, but the probability bound comes from the conformal guarantee on the calibration set, not from the construction. Lemma 3 is cited from the authors' prior work, but it is an elementary Minkowski-sum argument (σ ∈ F(s0) ⊕ PE with F(s0) ∈ ¯X and PE ∈ δX) and is not an unverified self-citation that forces the result. The comparison experiments use external simulators (quadcopter and powertrain) and the NNV reachability toolbox. The main weakness is that Experiment 3 asserts τ=0.04 as a valid TV bound for a 20% covariance increase without estimating or verifying it; however, this is a missing empirical validation of an input assumption, not a reduction of the claimed prediction to a fitted parameter or to a self-citation. Therefore no circular step is present.
Assumptions & free parameters
free parameters (4)
- ω_j =
max over training dataset of |r_j| for each j
- PE_mean_q =
mean of prediction error vectors over training dataset per segment (Eq. 10)
- V_q =
eigenvectors of the training error covariance matrix per segment (Eq. 10)
- τ (TV bound) =
0.04 in Experiment 3; 0 in others
assumptions (5)
- standard math Robust conformal inference quantile formula (Eq. 4) is valid
- domain assumption TV(J_sim, J_real) ≤ τ for the given τ
- domain assumption Calibration trajectories are i.i.d. from D_sim
- domain assumption Initial state distribution W has support on the compact set I
- domain assumption NNV star reachability gives a sound over-approximation of the ReLU network image
Cite this review
Pith. "Pith review of PCA-DDReach: Efficient Statistical Reachability Analysis of Stochastic Dynamical Systems via Principal Component Analysis." pith.science (2026). https://pith.science/paper/CIPJM6WL
@misc{pith2026250514935,
author = {Pith},
title = {Pith review of: PCA-DDReach: Efficient Statistical Reachability Analysis of Stochastic Dynamical Systems via Principal Component Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/CIPJM6WL}},
note = {Machine review of arXiv:2505.14935}
}
read the original abstract
This study presents a scalable data-driven algorithm designed to efficiently address the challenging problem of reachability analysis. Analysis of cyber-physical systems (CPS) relies typically on parametric physical models of dynamical systems. However, identifying parametric physical models for complex CPS is challenging due to their complexity, uncertainty, and variability, often rendering them as black-box oracles. As an alternative, one can treat these complex systems as black-box models and use trajectory data sampled from the system (e.g., from high-fidelity simulators or the real system) along with machine learning techniques to learn models that approximate the underlying dynamics. However, these machine learning models can be inaccurate, highlighting the need for statistical tools to quantify errors. Recent advancements in the field include the incorporation of statistical uncertainty quantification tools such as conformal inference (CI) that can provide probabilistic reachable sets with provable guarantees. Recent work has even highlighted the ability of these tools to address the case where the distribution of trajectories sampled during training time are different from the distribution of trajectories encountered during deployment time. However, accounting for such distribution shifts typically results in more conservative guarantees. This is undesirable in practice and motivates us to present techniques that can reduce conservatism. Here, we propose a new approach that reduces conservatism and improves scalability by combining conformal inference with Principal Component Analysis (PCA). We show the effectiveness of our technique on various case studies, including a 12-dimensional quadcopter and a 27-dimensional hybrid system known as the powertrain.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Avoiding geometric intersection operations in reachability analysis of hybrid systems
Matthias Althoff and Bruce H Krogh. Avoiding geometric intersection operations in reachability analysis of hybrid systems. In Proceedings of the 15th ACM international conference on Hybrid Systems: Computation and Control, pages 45--54, 2012
work page 2012
-
[2]
Simulation-equivalent reachability of large linear systems with inputs
Stanley Bak and Parasara Sridhar Duggirala. Simulation-equivalent reachability of large linear systems with inputs. In International Conference on Computer Aided Verification, pages 401--420. Springer, 2017
work page 2017
-
[3]
Luca Bortolussi, Francesca Cairoli, Nicola Paoletti, Scott A Smolka, and Scott D Stoller. Neural predictive monitoring. In Runtime Verification: 19th International Conference, RV 2019, Porto, Portugal, October 8--11, 2019, Proceedings 19, pages 129--147. Springer, 2019
work page 2019
-
[4]
Robust validation: Confident predictions even when distributions shift
Maxime Cauchois, Suyash Gupta, Alnur Ali, and John C Duchi. Robust validation: Confident predictions even when distributions shift. Journal of the American Statistical Association, pages 1--66, 2024
work page 2024
-
[5]
Conformal prediction regions for time series using linear complementarity programming
Matthew Cleaveland, Insup Lee, George J Pappas, and Lars Lindemann. Conformal prediction regions for time series using linear complementarity programming. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20984--20992, 2024
work page 2024
-
[6]
Alex Devonport and Murat Arcak. Data-driven reachable set computation using adaptive gaussian process classification and monte carlo methods. In Proc. of ACC, pages 2629--2634, 2020 a
work page 2020
-
[7]
Estimating reachable sets with scenario optimization
Alex Devonport and Murat Arcak. Estimating reachable sets with scenario optimization. In Learning for dynamics and control, pages 75--84. PMLR, 2020 b
work page 2020
-
[8]
Data-driven reachability analysis with christoffel functions
Alex Devonport, Forest Yang, Laurent El Ghaoui, and Murat Arcak. Data-driven reachability analysis with christoffel functions. In Proc. of CDC, pages 5067--5072, 2021
work page 2021
Show all 28 references
-
[9]
Nonconvex scenario optimization for data-driven reachability
Elizabeth Dietrich, Alex Devonport, and Murat Arcak. Nonconvex scenario optimization for data-driven reachability. In 6th Annual Learning for Dynamics & Control Conference, pages 514--527. PMLR, 2024
2024
-
[10]
Dryvr: Data-driven verification and compositional reasoning for automotive systems
Chuchu Fan, Bolun Qi, Sayan Mitra, and Mahesh Viswanathan. Dryvr: Data-driven verification and compositional reasoning for automotive systems. In International Conference on Computer Aided Verification, pages 441--461. Springer, 2017
2017
-
[11]
A general safety framework for learning-based control in uncertain robotic systems
Jaime F Fisac, Anayo K Akametalu, Melanie N Zeilinger, Shahab Kaynama, Jeremy Gillula, and Claire J Tomlin. A general safety framework for learning-based control in uncertain robotic systems. IEEE Transactions on Automatic Control, 64 0 (7): 0 2737--2752, 2018
2018
-
[12]
Scaling learning-based policy optimization for temporal logic tasks by controller network dropout
Navid Hashemi, Bardh Hoxha, Danil Prokhorov, Georgios Fainekos, and Jyotirmoy V Deshmukh. Scaling learning-based policy optimization for temporal logic tasks by controller network dropout. ACM Transactions on Cyber-Physical Systems, 8 0 (4): 0 1--28, 2024 a
2024
-
[13]
Statistical reachability analysis of stochastic cyber-physical systems under distribution shift
Navid Hashemi, Lars Lindemann, and Jyotirmoy V Deshmukh. Statistical reachability analysis of stochastic cyber-physical systems under distribution shift. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 43 0 (11): 0 4250--4261, 2024 b
2024
-
[14]
Intelligent computation of reachability sets for space missions
Erik Komendera, Daniel Scheeres, and Elizabeth Bradley. Intelligent computation of reachability sets for space missions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 2299--2304, 2012
2012
-
[15]
The empirical christoffel function with applications in data analysis
Jean B Lasserre and Edouard Pauwels. The empirical christoffel function with applications in data analysis. Advances in Computational Mathematics, 45: 0 1439--1468, 2019
2019
-
[16]
Safe planning in dynamic environments using conformal prediction
Lars Lindemann, Matthew Cleaveland, Gihyun Shim, and George J Pappas. Safe planning in dynamic environments using conformal prediction. IEEE Robotics and Automation Letters, 2023
2023
-
[17]
Formal verification and control with conformal prediction
Lars Lindemann, Yiqi Zhao, Xinyi Yu, George J Pappas, and Jyotirmoy V Deshmukh. Formal verification and control with conformal prediction. arXiv preprint arXiv:2409.00536, 2024
2024 arXiv
-
[18]
Semi-algebraic approximation using christoffel--darboux kernel
Swann Marx, Edouard Pauwels, Tillmann Weisser, Didier Henrion, and Jean Bernard Lasserre. Semi-algebraic approximation using christoffel--darboux kernel. Constructive Approximation, pages 1--39, 2021
2021
-
[19]
Verification of neural-network control systems by integrating taylor models and zonotopes
Christian Schilling, Marcelo Forets, and Sebasti \'a n Guadalupe. Verification of neural-network control systems by integrating taylor models and zonotopes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8169--8177, 2022
2022
-
[20]
Pac-bayes generalization certificates for learned inductive conformal prediction
Apoorva Sharma, Sushant Veer, Asher Hancock, Heng Yang, Marco Pavone, and Anirudha Majumdar. Pac-bayes generalization certificates for learned inductive conformal prediction. Advances in Neural Information Processing Systems, 36, 2024
2024
-
[21]
Introduction to nonparametric regression
Kunio Takezawa. Introduction to nonparametric regression. John Wiley & Sons, 2005
2005
-
[22]
Data-driven reachability using christoffel functions and conformal prediction
Abdelmouaiz Tebjou, Goran Frehse, et al. Data-driven reachability using christoffel functions and conformal prediction. In Conformal and Probabilistic Prediction with Applications, pages 194--213. PMLR, 2023
2023
-
[23]
Scalable safe long-horizon planning in dynamic environments leveraging conformal prediction and temporal correlations
Sander Tonkens, Sophia Sun, Rose Yu, and Sylvia Herbert. Scalable safe long-horizon planning in dynamic environments leveraging conformal prediction and temporal correlations. In Long-Term Human Motion Prediction Workshop, International Conference on Robotics and Automation, 2023
2023
-
[24]
Nnv: the neural network verification tool for deep neural networks and learning-enabled cyber-physical systems
Hoang-Dung Tran, Xiaodong Yang, Diego Manzanas Lopez, Patrick Musau, Luan Viet Nguyen, Weiming Xiang, Stanley Bak, and Taylor T Johnson. Nnv: the neural network verification tool for deep neural networks and learning-enabled cyber-physical systems. In Proc. of CAV, pages 3--17, 2020
2020
-
[25]
Multi-modal conformal prediction regions by optimizing convex shape templates
Renukanandan Tumu, Matthew Cleaveland, Rahul Mangharam, George Pappas, and Lars Lindemann. Multi-modal conformal prediction regions by optimizing convex shape templates. In 6th Annual Learning for Dynamics & Control Conference, pages 1343--1356. PMLR, 2024
2024
-
[26]
Conditional validity of inductive conformal predictors
Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learning, pages 475--490. PMLR, 2012
2012
-
[27]
Forking uncertainties: Reliable prediction and model predictive control with sequence models via conformal risk control
Matteo Zecchin, Sangwoo Park, and Osvaldo Simeone. Forking uncertainties: Reliable prediction and model predictive control with sequence models via conformal risk control. IEEE Journal on Selected Areas in Information Theory, 2024
2024
-
[28]
Reachability analysis of neural network control systems
Chi Zhang, Wenjie Ruan, and Peipei Xu. Reachability analysis of neural network control systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15287--15295, 2023
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.