REVIEW 2 major objections 5 minor 27 references
Clustering Tails in High Dimension
T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A rank-based iterative algorithm clusters high-dimensional variables by their extreme value index, and the partition it returns is provably consistent—no per-variable index estimation and no assumptions on dependence structure.
desk verdict A promising new tail-clustering method whose main theorems are vacuous as printed due to a reciprocal sign error in condition (C2); fixable, but must be fixed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a comparison between two rank-based quantities after self-scaling. Each variable $X^{(j)}$ is divided by its $(n-k^*)$-th order statistic, so the high-order statistics of the scaled variable behave like $x^{\gamma_j}$. At iteration $\ell$ the algorithm pools the scaled observations of the remaining variables and sets the threshold $u_\ell$ to the $(k \cdot |I_\ell|)$-th upper order statistic of that pool; a variable joins the current group if its own $\lfloor \beta k \rfloor$-th upper order statistic is at least $u_\ell$. Because the heaviest-tailed variables dominate the pooled tail, their quantiles clear the threshold while lighter-tailed variables fall short; the proof controls the ratio of these quantiles, and condition (C2) makes the separating ratio $(k^*/k)^{-\Delta^{(\ell)}} |I_\ell|^{\gamma^{(\ell)}}$ vanish.
What would settle it
Simulate two groups with extreme value indices $\gamma_1 = 1$ and $\gamma_2 = 0.5$ (so $\Delta = 0.5$), set $p = n^2$, and choose $k$ and $k^*$ as in Remark 2 (e.g., $k = 3\log_{1.05} p$, $k^* = n^{0.98}$); then $p(k^*/k)^\Delta$ does not vanish and condition (C2) fails. If the algorithm still recovers the true partition with probability close to 1 in repeated simulations, the stated dimension bound is not needed; if accuracy drops noticeably as $p$ grows, that confirms the bound is load-bearing.
Extended reading notes
Core claim
The paper's central claim is that the partition $\hat{\tau}$ returned by the proposed iterative algorithm equals the true grouping $\tau$ of variables by common extreme value index with probability at least $1 - C_1 p g \exp(-C_2 k)$ for sufficiently large $n$, under conditions (C1) and (C2). When $\log(p)/k \to 0$, the probability tends to 1. The consistency holds without any assumption on the dependence structure among the $p$ variables, including no sparsity or graph constraints; the only conditions are a uniform regular-variation assumption on the marginals and the dimension bound $p(k^*/k)^\Delta \to 0$, where $\Delta$ is the minimum relative gap between consecutive group indices. The same guarantee extends to the case where the number of groups is unknown.
Load-bearing premise
The proof needs the number of variables $p$ to grow slowly enough relative to the separation between group tail indices; when $p$ grows too fast for the method's tuning parameters, the quantile comparison that separates the clusters can break down.
Editorial extensions
If this is right
- Before pooling tail information across variables, the algorithm identifies which variables share a common extreme value index, making pooled Hill-type estimation valid in heterogeneous panels.
- The method works without knowing the dependence structure, so it applies when variables have arbitrary tail dependence, including strong dependence.
- The number of clusters is determined adaptively, avoiding the need to pre-specify $g$.
- Simulations suggest that clustering first and then averaging group-wise Hill estimates reduces mean squared error compared with per-variable Hill estimation or two-step tail $k$-means.
- The algorithm is computationally cheap because it uses only order statistics and comparisons, which is advantageous for large $p$.
Reading between the lines
- The same self-scaling-plus-pooled-threshold idea could be adapted to cluster by second-order tail parameters or by extreme quantiles at a fixed level, not just the index; the paper does not pursue this.
- A natural extension would be a data-driven selection of $k$ and $k^*$ instead of the heuristic in Remark 2; the theory only says that some choices exist.
- In distributed settings where raw data cannot be pooled, the algorithm needs only per-variable order statistics, so a communication-efficient variant could be developed, although the paper does not address this.
- The exponentially fast consistency bound suggests that multiple thresholds could be used to estimate $\Delta$ itself, turning the algorithm into a tool for gap estimation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a sequential clustering procedure for high-dimensional random vectors, grouping variables according to their extreme value indices. Each marginal is first self-scaled by a high order statistic, and the algorithm then iteratively extracts the group with the largest remaining index by comparing per-variable upper order statistics with a pooled upper order statistic. Theorems 1 and 2 claim consistency when the number of groups is known, and Theorem 3 makes the same claim when it is unknown, under a uniform regular-variation condition and a dimension/intermediate-sequence condition. A simulation study and a currency exchange-rate application are also presented.
Significance. The intended contribution is potentially valuable: the method avoids pre-estimating each marginal tail index, does not require the number of clusters when g is unknown, imposes no assumptions on the dependence structure across variables, and is designed for high dimensions. The proof structure is transparent and uses standard exponential inequalities, and the simulations cover several marginal models and dependence structures. However, the central condition (C2), Eq. (3), is stated in an unsatisfiable form, which makes Theorems 1–3 vacuous as printed; the proof itself shows the reciprocal condition that is actually needed. In addition, Algorithm 2 can fail to terminate in finite samples. Both issues are correctable, but they must be fixed before the consistency claims can be accepted.
major comments (2)
- [Section 2.1, Eq. (3)]
- [Section 2.2, Algorithm 2]
minor comments (5)
- [Section 3, Algorithm 3]
- [Section 5, proof of Theorem 1]
- [Section 3]
- [Remark 2 vs Section 3]
- [Figure S3]
Circularity Check
No significant circularity: the clustering consistency result is derived from regular-variation and intermediate-sequence assumptions, not from fitted inputs, though Eq. (3) as printed appears unsatisfiable (a correctness concern, not circularity).
full rationale
The paper's claimed derivation is not circular. Algorithm 1 constructs clusters by comparing each variable's self-scaled upper order statistic with a pooled upper order statistic; the threshold is recomputed from the current candidate set and no parameter is fitted to the true partition tau. Consistency in Theorems 1-2 follows from uniform regular variation (C1), the intermediate-sequence conditions (C2), and exponential bounds for order statistics (Lemma A2); the proof establishes separation of the relevant ratios on a high-probability set, so the conclusion is not assumed in the input. Self-citations, such as Chen and Zhou (2024) for uniform regular-variation tools and Chen et al. (2022, 2025) for distributed Hill estimation, are supporting technical ingredients rather than load-bearing uniqueness or ansatz claims. Two non-circular concerns should be flagged: as printed, Eq. (3) p(k*/k)^Delta -> 0 is unsatisfiable because k/k* -> 0 and Delta > 0 imply (k*/k)^Delta -> infinity while p >= 1; the proof of relation (5) in Section 5 actually requires the reciprocal condition p(k/k*)^Delta -> 0, equivalently |I_l| (k*/k)^{-Delta} -> 0. This makes the stated assumptions vacuous and is a sign-error/correctness issue, not a circularity. Additionally, the proof of Theorem 3 is omitted as analogous. These issues do not change the circularity verdict.
Assumptions & free parameters
free parameters (4)
- k =
floor(3 log_{1.05} p) in Remark 2
- k* =
floor(n_0^{0.98}), n_0 = minimum positive count across dimensions
- beta =
min(2 (k*/k)^{-1} p + 0.5, 0.9)
- g =
user-specified for Algorithm 1; inferred in Algorithm 2
assumptions (6)
- domain assumption Marginals F_j are in the max-domain of attraction of a GEV distribution with positive extreme value index gamma_j > 0, as in Eq. (1).
- domain assumption Uniform regular variation condition (C1): the survival ratio converges to x^{-1/gamma_j} uniformly over j and x > 1.
- domain assumption Condition (C2): k, k* are intermediate sequences with k/k* -> 0, k*/n -> 0, and p(k*/k)^Delta -> 0, where Delta is the minimum relative gap between adjacent group indices.
- domain assumption The true partition tau exists with distinct ordered indices gamma^(1) > ... > gamma^(g) > 0.
- domain assumption The self-scaling denominator X^(j)_{n-k*:n} is positive for all j, with Remark 2 using the minimum positive count n_0.
- standard math Bernstein's inequality from Shorack and Wellner (1986), Lemma A1.
Cite this review
Pith. "Pith review of Clustering Tails in High Dimension." pith.science (2026). https://pith.science/paper/URXKOYTY
@misc{pith2026250619414,
author = {Pith},
title = {Pith review of: Clustering Tails in High Dimension},
year = {2026},
howpublished = {\url{https://pith.science/paper/URXKOYTY}},
note = {Machine review of arXiv:2506.19414}
}
read the original abstract
One potential solution to combat the scarcity of tail observations in extreme value analysis is to integrate information from multiple datasets sharing similar tail properties, for instance, a common extreme value index. In other words, for a multivariate dataset, we intend to group dimensions into clusters first, before applying any pooling techniques. This paper addresses the clustering problem for a high dimensional dataset, according to their extreme value indices. We propose an iterative clustering procedure that sequentially partitions the variables into groups, ordered from the heaviest-tailed to the lightesttailed distributions. At each step, our method identifies and extracts a group of variables that share the highest extreme value index among the remaining ones. This approach differs fundamentally from conventional clustering methods such as using pre-estimated extreme value indices in a two-step clustering method. We show the consistency property of the proposed algorithm and demonstrate its finite-sample performance using a simulation study and a real data application.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Battey, H., Fan, J., Liu, H., Lu, J., and Zhu, Z. (2018). Distributed testing and estimation under sparse high dimensional models. Annals of statistics , 46(3):1352
work page 2018
-
[2]
Beirlant, J., Goegebeur, Y., Segers, J., and Teugels, J. L. (2004). Statistics of Extremes: Theory and Applications . John Wiley & Sons
work page 2004
-
[3]
Cai, T., Liu, M., and Xia, Y. (2022). Individual data protected integrative regression analysis of high-dimensional heterogeneous data. Journal of the American Statistical Association , 117(540):2105--2119
work page 2022
-
[4]
Cao, H., Gao, J., Shao, Y., Sriram, T., Wang, W., Wen, F., and Zhang, T. (2024). Tail index estimation for tail adversarial stable time series with an application to high-dimensional tail clustering. Journal of Time Series Analysis
work page 2024
-
[5]
Chen, L., Li, D., and Zhou, C. (2022). Distributed inference for the extreme value index. Biometrika , 109(1):257--264
work page 2022
-
[6]
Chen, L., Li, D., and Zhou, C. (2025). Distributed inference for tail risk. Statistica Sinica
work page 2025
-
[7]
Chen, L. and Zhou, C. (2024). High dimensional inference for extreme value indices. arXiv preprint arXiv:2407.20491
arXiv 2024
-
[8]
Daouia, A., Padoan, S. A., and Stupfler, G. (2024). Optimal weighted pooling for inference about the tail index and extreme quantiles. Bernoulli , 30(2):1287--1312
work page 2024
Show all 27 references
-
[9]
Davison, A. C. and Smith, R. L. (1990). Models for exceedances over high thresholds. Journal of the Royal Statistical Society Series B: Statistical Methodology , 52(3):393--425
1990
-
[10]
de Carvalho, M., Huser, R., and Rubio, R. (2023). Similarity-based clustering for patterns of extreme values. Stat , 12(1):e560
2023
-
[11]
and Ferreira, A
de Haan, L. and Ferreira, A. (2006). Extreme Value Theory: an Introduction . Springer
2006
-
[12]
H., Ferreira, A., de Haan, L., Neves, C., and Zhou, C
Einmahl, J. H., Ferreira, A., de Haan, L., Neves, C., and Zhou, C. (2022). Spatial dependence and space--time trend in extreme events. Annals of Statistics , 50(1):30--52
2022
-
[13]
Engelke, S., Lalancette, M., and Volgushev, S. (2021). Learning extremal graphical structures in high dimensions. arXiv preprint arXiv:2111.00840
2021
-
[14]
and Volgushev, S
Engelke, S. and Volgushev, S. (2022). Structure learning for extremal tree models. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(5):2055--2087
2022
-
[15]
Hill, B. M. (1975). A simple general approach to inference about the tail of a distribution. Annals of Statistics , 3(5):1163--1174
1975
-
[16]
and Oesting, M
Lederer, J. and Oesting, M. (2023). Extremes in high dimensions: Methods and scalable algorithms. arXiv preprint arXiv:2303.04258
2023 arXiv
-
[17]
T., and Li, H
Li, S., Cai, T. T., and Li, H. (2022). Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(1):149--173
2022
-
[18]
and Zhang, L
Li, S. and Zhang, L. (2025). Multi-dimensional domain generalization with low-rank structures. Journal of the American Statistical Association , pages 1--13
2025
-
[19]
Liang, X., Li, S., Zhang, S., Huang, H., and Chen, S. X. (2016). Pm2. 5 data reliability, consistency, and air quality assessment in five chinese cities. Journal of Geophysical Research: Atmospheres , 121(17):10--220
2016
-
[20]
Resnick, S. I. (2008). Extreme Values, Regular Variation, and Point Processes . Springer Science & Business Media
2008
-
[21]
Shorack, G. R. and Wellner, J. A. (1986). Empirical Processes with Applications to Statistics . Wiley
1986
-
[22]
and Feng, Y
Tian, Y. and Feng, Y. (2023). Transfer learning under high-dimensional generalized linear models. Journal of the American Statistical Association , 118(544):2684--2697
2023
-
[23]
and Chao, S.-K
Volgushev, S. and Chao, S.-K. (2019). Distributed inference for quantile regression processes. Annals of statistics , 47(3):1634--1662
2019
-
[24]
and Zhou, C
Wan, P. and Zhou, C. (2023). Graphical lasso for extremes. arXiv preprint arXiv:2307.15004
2023 arXiv
-
[25]
J., Lin, Y., and Schaumburg, J
Wang, C., Cai, J. J., Lin, Y., and Schaumburg, J. (2025). Clustering extreme value indices in large panels. Technical report, Tinbergen Institute Discussion Paper
2025
-
[26]
and Qu, A
Xu, Q. and Qu, A. (2025). Representation retrieval learning for heterogeneous data integration. arXiv preprint arXiv:2503.09494
2025
-
[27]
Zhang, W., Deng, L., Zhang, L., and Wu, D. (2022). A survey on negative transfer. IEEE/CAA Journal of Automatica Sinica , 10(2):305--329
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.