REVIEW 4 major objections 7 minor 45 references
Hierarchical Superpixel Segmentation via Structural Information Theory
T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that SIT-HSS, built on structural information theory, outperforms nine existing unsupervised superpixel methods on three benchmarks.
desk verdict New SE-based superpixel pipeline with plausible gains; per-dataset τ tuning and missing error bars keep the SOTA claim one notch short of proven, but it deserves serious review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are the $1$-dimensional and $2$-dimensional structural entropies of a weighted pixel graph. The $1$D SE, $H^{(1)}_r(G) = -\sum_{i=1}^{|V|} (d_i / V^r_G) \log(d_i / V^r_G)$, is the entropy of the degree distribution; increasing the search radius $r$ adds edges to farther neighbors and raises this entropy, and Eq. (3.6) stops the expansion at the first radius where the increment falls below a threshold $\tau$. The $2$D SE, $H^{(2)}_P(G) = -\sum_{p_i \in P} \left[ (g_{p_i}/V_G) \log(V_{p_i}/V_G) + \sum_{j \in p_i} (d_j / V_G) \log(d_j / V_{p_i}) \right]$, scores a partition by cut weights and volumes, and the merging operator greedily selects the adjacent pair with the largest $\Delta H^{(2)}_{p_i,p_j}$ (Eq. 3.8). The coupling of the two entropy notions is the core mechanism: the $1$D version decides what information enters the graph, the $2$D version decides how to group it.
What would settle it
A decisive test is to take textured images whose one-dimensional entropy curve rises smoothly, sweep $\tau$ from very small to very large, and compare Boundary Recall; if no threshold reproduces the reported quality, or if fixing $r=1$ already gives the same quality, then the graph-construction mechanism is not carrying the improvement.
Extended reading notes
Core claim
The paper's central claim is that structural entropy can serve as the single organizing principle for unsupervised superpixel segmentation. SIT-HSS first constructs a weighted pixel graph by expanding each pixel's neighborhood radius until the one-dimensional structural entropy—the entropy of the graph's degree distribution—stops growing by more than a per-dataset threshold; this is intended to capture relationships between non-adjacent pixels while keeping the graph sparse. It then partitions the graph by repeatedly merging adjacent clusters whose union gives the largest decrease in two-dimensional structural entropy, until the requested number of superpixels $K$ is reached. On BSDS500, SBD, and PASCAL-S at $K=600$, the paper reports that SIT-HSS achieves the best ASA, BR, and UE among all nine baselines and is best or second-best on EV.
Load-bearing premise
The load-bearing premise is that a fixed rule—stop expanding the pixel neighborhood at the first radius where the one-dimensional entropy increase drops below a threshold—yields a graph that retains enough information for accurate superpixel boundaries, and that this threshold can be tuned per dataset rather than derived from image content.
Editorial extensions
If this is right
- If the reported results hold, downstream tasks that consume superpixels inherit better boundary recall and lower undersegmentation error, which should reduce error propagation in segmentation, saliency, and classification pipelines.
- The method yields a full merge hierarchy, so any intermediate merge count is a valid coarse-to-fine representation; a single run covers multiple target sizes $K$.
- The algorithm is unsupervised and needs only three parameters ($t$, $\tau$, $K$), so it applies where labeled data are unavailable.
- At $K=600$ the reported runtime is within about a tenth of a second of the fastest baselines per image and several times faster than other graph-based methods, so the quality gain does not come with a large speed penalty.
Reading between the lines
- Pith inference: The plateau-stopping rule in Eq. (3.6) could be replaced by a curvature-based or model-selection criterion, which might remove the per-dataset tuning of $\tau$ and make graph construction adaptive.
- Pith inference: Because SIT-HSS produces a complete hierarchy of merges, the same run can serve any downstream method that needs superpixels at several resolutions, making the hierarchy itself a reusable asset.
- Pith inference: The graph-construction strategy is not tied to raw color and position features; substituting learned embeddings would test whether structural entropy remains a useful organizing principle for deep representations, an experiment the paper does not run.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SIT-HSS, an unsupervised hierarchical superpixel segmentation method built on structural information theory. The method constructs a pixel graph by progressively adding edges to an expanding neighborhood until the increment of 1D structural entropy falls below a threshold tau, and then partitions the graph by greedily merging adjacent superpixel pairs that most reduce the 2D structural entropy, until a target number K of superpixels is reached. The authors report experiments on BSDS500, SBD, and PASCAL-S showing that SIT-HSS outperforms nine baselines on ASA, BR, and UE at K=600, with competitive runtime. The source code is made publicly available.
Significance. If the reported results are robust, the paper offers a useful new application of structural information theory to superpixel segmentation and provides a transparent, interpretable alternative to black-box deep superpixel methods. The derivation of the 2D structural-entropy merging rule in Eq. (3.8) is algebraically consistent with Definition 2, the pseudocode is implementable, and the release of source code is a concrete strength. The main significance hinges on the empirical state-of-the-art claim, which currently rests on a per-dataset-tuned plateau heuristic and on single-run comparisons without statistical support; those issues need to be addressed before the SOTA claim can be considered established.
major comments (4)
- [Section 4.1, Table 1] The radius r is selected solely by the first 1D structural-entropy increment below the threshold tau, but the paper does not establish the assumed monotone-plateau behavior of H_r^(1)(G), nor does it report the distribution of the selected r values across the three datasets. The threshold tau is tuned per dataset (2e-7 for BSDS500 and PASCAL-S, 1e-6 for SBD), and the authors' own sensitivity analysis shows nontrivial metric swings with tau (for example, on PASCAL-S the EV changes from 90.81 at tau=1e-7 to 87.86 at tau=4e-7). Since Eq. (3.6) determines the graph on which all subsequent merging operates, the state-of-the-art claim currently rests on an unvalidated, dataset-specific plateau detector. Please add diagnostics of the H_r^(1) curves, the selected r distribution, and either an ablation with a fixed tau or a principled automatic selection rule.
- [Section 4.1, Table 1] The headline comparison reports a single number per method, per dataset, and per metric, with no error bars, no repeated trials, and no significance tests. Several reported margins are very small (for example, SBD ASA: 95.250 for SIT-HSS vs. 95.203 for ETPS), and on PASCAL-S EV the method is not the best (ETPS: 91.052 vs. SIT-HSS: 90.870). Given that the authors tune tau on the same datasets used for the headline comparison, the state-of-the-art claim needs at least confidence intervals over images, bootstrap estimates, or pairwise significance tests, together with a statement of whether the baselines' hyperparameters were tuned on the same datasets.
- [Section 5 and Table 1] The comparison omits the hierarchical superpixel methods SH, CRTrees, and HHTS, which are discussed in Related Work and are the most directly relevant competitors for a hierarchical method such as SIT-HSS. If these methods cannot be compared because of code availability or different experimental conditions, the SOTA claim should be narrowed accordingly, or the missing comparisons should be provided.
- [Section 3.4] The claimed O(|E|) time complexity is not fully justified. In early merging iterations, the product |P| * m (number of superpixels times average number of adjacent superpixels) can exceed |E|, so the inequality l2 * |P| * m <= O(|E|) needs a proof rather than a statement. The empirical convergence plot in Fig. 11 supports practical efficiency, but the worst-case bound should be stated more cautiously or derived rigorously.
minor comments (7)
- [Eq. (3.6)] The notation 'arg min_i { i | ... }' should be 'min { i | ... }' because the expression denotes the smallest index, not an argument minimizing a function.
- [Algorithm 1 vs. Eq. (3.6)] The pseudocode uses the condition 'H_r^(1) - H_{r-1}^(1) <= tau' while Eq. (3.6) uses '< tau'; the two should be made consistent.
- [Eq. (3.3)] The Frobenius norm is invoked for vectors; an ordinary Euclidean norm would be the standard notation for color and position differences.
- [Section 4.1, Table 1] The sentence 'outperforms all baselines across both three datasets' is contradicted by the PASCAL-S EV column, where ETPS is higher than SIT-HSS; the claim should be revised to state that SIT-HSS is best on some metrics and runner-up on EV for that dataset.
- [Table 1 and text] The baseline name 'DRW' is inconsistently written as 'DR W' in Table 1 and elsewhere; please unify the spelling.
- [Algorithm 1] The 'Break' on line 13 is ambiguous: it should be clear whether it exits the r-loop, the pixel loop, or the neighbor loop, and the indentation should make the control flow explicit.
- [Figure 5] The graph embedding and encoding-tree visualizations are difficult to read at the current resolution; higher-resolution panels with labeled axes and clearer tree nodes would strengthen the interpretability claim.
Circularity Check
No significant circularity: the SOTA claim is an external benchmark comparison, and the graph-construction and partitioning heuristics are measured, not derived from the benchmark.
full rationale
The central claim (Section 4.1, Table 1) is an empirical comparison of SIT-HSS against nine external baselines on BSDS500, SBD, and PASCAL-S. No equation in the paper derives ASA, BR, UE, or EV from the method's own definitions by construction: Eq. (3.6) is a plateau-selection heuristic for choosing the neighborhood radius, and Eqs. (3.7)-(3.9) define a greedy 2D-SE merging objective, but the reported metrics are measured outcomes, not analytic consequences of those equations. The structural entropy framework is cited to Li and Pan [14], an external source with no author overlap, so the core theory is not imported from a self-citation chain. The many citations to the authors' earlier structural-entropy papers (e.g., [4, 19, 25, 33, 36-39, 44]) appear in the preliminaries and related work as examples of successful applications; they are motivational and do not carry the benchmark claim. The only circularity-adjacent issue is that t and tau are tuned on the same datasets used for the headline comparison (Section 4, Implementation Details; Figs. 6, 9, 10), which is a potential overfitting and reproducibility concern, not a logical reduction of the result to its inputs. The fragility of the untested monotone-plateau assumption in Eq. (3.6) is a robustness concern, not circularity.
Assumptions & free parameters
free parameters (2)
- t (weight normalization parameter) =
0.1
- tau (1D SE increment threshold) =
2e-7 (BSDS500, PASCAL-S), 1e-6 (SBD)
assumptions (4)
- standard math Structural entropy definitions and the MERGE operator from Li and Pan (2016) are adopted without re-derivation.
- domain assumption Pixel similarity is adequately captured by the product of squared color difference and spatial distance, normalized by mean edge distance (Eqs. 3.3-3.4).
- ad hoc to paper The first radius where the 1D SE increment falls below tau yields a graph that retains information relevant to superpixel boundaries and compactness.
- domain assumption Greedy merging of the adjacent pair with maximum 2D SE decrease reaches a good segmentation at target K.
Cite this review
Pith. "Pith review of Hierarchical Superpixel Segmentation via Structural Information Theory." pith.science (2026). https://pith.science/paper/OESS4XZG
@misc{pith2026250107069,
author = {Pith},
title = {Pith review of: Hierarchical Superpixel Segmentation via Structural Information Theory},
year = {2026},
howpublished = {\url{https://pith.science/paper/OESS4XZG}},
note = {Machine review of arXiv:2501.07069}
}
read the original abstract
Superpixel segmentation is a foundation for many higher-level computer vision tasks, such as image segmentation, object recognition, and scene understanding. Existing graph-based superpixel segmentation methods typically concentrate on the relationships between a given pixel and its directly adjacent pixels while overlooking the influence of non-adjacent pixels. These approaches do not fully leverage the global information in the graph, leading to suboptimal segmentation quality. To address this limitation, we present SIT-HSS, a hierarchical superpixel segmentation method based on structural information theory. Specifically, we first design a novel graph construction strategy that incrementally explores the pixel neighborhood to add edges based on 1-dimensional structural entropy (1D SE). This strategy maximizes the retention of graph information while avoiding an overly complex graph structure. Then, we design a new 2D SE-guided hierarchical graph partitioning method, which iteratively merges pixel clusters layer by layer to reduce the graph's 2D SE until a predefined segmentation scale is achieved. Experimental results on three benchmark datasets demonstrate that the SIT-HSS performs better than state-of-the-art unsupervised superpixel segmentation algorithms. The source code is available at \url{https://github.com/SELGroup/SIT-HSS}.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Ssstrunk, Slic superpixels compared to state-of-the-art superpixel methods , IEEE TPAMI, 34 (2012), pp. 2274–2282
work page 2012
-
[2]
R. Achanta and S. Susstrunk, Superpixels and poly- gons using simple non-iterative clustering , in Proceed- ings of the CVPR, 2017, pp. 4651–4660
work page 2017
-
[3]
P. Arbelez, M. Maire, C. Fowlkes, and J. Ma- lik, Contour detection and hierarchical image segmen- tation, IEEE TPAMI, 33 (2011), pp. 898–916
work page 2011
-
[4]
Y. Cao, H. Peng, Z. Yu, and S. Y. Philip, Hierar- chical and incremental structural entropy minimization for unsupervised social event detection , in Proceedings of the AAAI, vol. 38, 2024, pp. 8255–8264
work page 2024
-
[5]
T. V. Chang, S. Seibt, and B. von Rymon Lip- inski, Hierarchical histogram threshold segmentation- auto-terminating high-detail oversegmentation , in Pro- ceedings of the CVPR, 2024, pp. 3195–3204
work page 2024
-
[6]
J. Chen, Z. Li, and B. Huang , Linear spectral clustering superpixel , IEEE TIP, 26 (2017), pp. 3317– 3330
work page 2017
- [7]
- [8]
Show all 45 references
-
[9]
Huang, Y.-H
T.-K. Huang, Y.-H. W ang, T.-K. Lin, and Y.- Y. Chuang, A robust automatic object segmentation method for 3d printing , in Proceddings of the ICME, IEEE, 2016, pp. 1–6
2016
-
[10]
V. Jain, S. C. Turaga, K. Briggman, M. Helm- staedter, W. Denk, and H. Seung, Learning to agglomerate superpixel hierarchies, Proceedings of the NeurIPS, 24 (2011), pp. 1–9
2011
-
[11]
X. Kang, L. Zhu, and A. Ming, Dynamic random walk for superpixel segmentation, IEEE TIP, 29 (2020), pp. 3871–3884
2020
-
[12]
S. G. Kobourov , Spring embedders and force directed graph drawing algorithms , arXiv preprint arXiv:1201.3011, (2012), pp. 1–23
2012 arXiv
-
[13]
T. Lei, X. Jia, Y. Zhang, S. Liu, H. Meng, and A. K. Nandi, Superpixel-based fast fuzzy c-means clus- tering for color image segmentation , IEEE Transac- tions on Fuzzy Systems, 27 (2018), pp. 1753–1766
2018
-
[14]
Li and Y
A. Li and Y. Pan, Structural information and dy- namical complexity of networks , IEEE TIT, 62 (2016), pp. 3290–3339
2016
-
[15]
H. Li, Y. Jia, R. Cong, W. Wu, S. T. W. Kwong, and C. Chen, Superpixel segmentation based on spa- tially constrained subspace clustering , IEEE TII, 17 (2020), pp. 7501–7512
2020
-
[16]
Y. Li, X. Hou, C. Koch, J. M. Rehg, and A. L. Yuille, The secrets of salient object segmentation , in Procedings of the CVPR, 2014, pp. 280–287
2014
-
[17]
M.-Y. Liu, O. Tuzel, S. Ramalingam, and R. Chellappa, Entropy rate superpixel segmentation , in CVPR, 2011, pp. 2097–2104
2011
-
[18]
T. C. Ng, S. K. Choy, S. Y. Lam, and K. W. Yu, Fuzzy superpixel-based image segmentation , Pattern Recognition, 134 (2023), p. 109045
2023
-
[19]
H. Peng, J. Zhang, X. Huang, Z. Hao, A. Li, Z. Yu, and P. S. Yu, Unsupervised social bot detection via structural information theory , ACM Transactions on Information Systems, 42, pp. 1–42
-
[20]
Y. Qiu, J. Mei, and J. Xu, Superpixel-wise contrast exploration for salient object detection , Knowledge- Based Systems, 292 (2024), p. 111617
2024
-
[21]
Ren and J
X. Ren and J. Malik, Learning a classification model for segmentation , Proceedings of the ICCV, (2003), pp. 10–17
2003
-
[22]
Sellars, A
P. Sellars, A. I. A viles-Rivero, and C.-B. Sch¨onlieb, Superpixel contracted graph-based learning for hyperspectral image classification, IEEE TGRS, 58 (2020), pp. 4180–4193
2020
-
[23]
J. Shen, X. Hao, Z. Liang, Y. Liu, W. W ang, and L. Shao, Real-time superpixel segmentation by dbscan clustering algorithm , IEEE TIP, 25 (2016), pp. 5933– 5942
2016
-
[24]
Shi and J
J. Shi and J. Malik, Normalized cuts and image segmentation, IEEE TPAMI, 22 (2000), pp. 888–905
2000
-
[25]
L. Sun, Z. Huang, H. Peng, Y. W ang, C. Liu, and S. Y. Philip, Lsenet: Lorentz structural entropy neural network for deep graph clustering , in Procedings of the ICML, 2024, pp. 1–12
2024
-
[26]
D. Tang, H. Fu, and X. Cao, Topology preserved regular superpixel, in Proceedings of the ICME, IEEE, 2012, pp. 765–768
2012
-
[27]
van den Bergh, X
M. van den Bergh, X. Boix, G. Roig, B. de Cap- itani, and L. V. Gool, Seeds: Superpixels extracted via energy-driven sampling , IJCV, 111 (2012), pp. 298 – 314
2012
-
[28]
Vedaldi and S
A. Vedaldi and S. Soatto, Quick shift and kernel methods for mode seeking, in Proceedings of the ECCV, Springer, 2008, pp. 705–718
2008
-
[29]
W ang, Y
Y. W ang, Y. Wei, X. Qian, L. Zhu, and Y. Yang, Ainet: Association implantation for superpixel segmen- tation, in Proceedings of the CVPR, 2021, pp. 7078– 7087
2021
-
[30]
X. Wei, Q. Yang, Y. Gong, N. Ahuja, and M.- H. Yang, Superpixel hierarchy, IEEE TIP, 27 (2018), pp. 4838–4849
2018
-
[31]
T. Yan, X. Huang, and Q. Zhao, Hierarchical su- Copyright © 2025 by SIAM Unauthorized reproduction of this article is prohibited perpixel segmentation by parallel crtrees labeling, IEEE TIP, 31 (2022), pp. 4719–4732
2022
-
[32]
F. Yang, Q. Sun, H. Jin, and Z. Zhou, Superpixel segmentation with fully convolutional networks , in Pro- ceedings of the CVPR, 2020, pp. 13961–13970
2020
-
[33]
R. Yang, H. Peng, A. Li, P. Li, C. Liu, and S. Y. Philip, Hierarchical abstracting graph kernel , IEEE Transactions on Knowledge and Data Engineering, (2024), pp. 724–738
2024
-
[34]
J. Yao, M. Boben, S. Fidler, and R. Urta- sun, Real-time coarse-to-fine topologically preserving segmentation, in Proceedings of the CVPR, 2015, pp. 2947–2955
2015
-
[35]
Y. Yuan, Z. Zhu, H. Yu, and W. Zhang, Watershed- based superpixels with global and local boundary march- ing, IEEE TIP, 29 (2020), pp. 7375–7388
2020
-
[36]
G. Zeng, H. Peng, A. Li, Z. Liu, C. Liu, S. Y. Philip, and L. He, Unsupervised skin lesion segmenta- tion via structural entropy minimization on multi-scale superpixel graphs, in Procedings of the ICDM, IEEE, 2023, pp. 768–777
2023
-
[37]
G. Zeng, H. Peng, A. Li, J. Wu, C. Liu, and S. Y. Philip, Scalable semi-supervised clustering via struc- tural entropy with different constraints, IEEE Transac- tions on Knowledge and Data Engineering, 37 (2025), pp. 478–492
2025
-
[38]
X. Zeng, H. Peng, and A. Li, Effective and stable role-based multi-agent collaboration by structural infor- mation principles, in Proceedings of the AAAI, vol. 37, 2023, pp. 11772–11780
2023
-
[39]
X. Zeng, H. Peng, and A. Li, Effective exploration based on the structural information principles , in Pro- ceedings of the NeurIPS, 2024, pp. 1–35
2024
-
[40]
Zhang, C
G. Zhang, C. Xue, and R. Zhang, Supernerf: High- precision 3d reconstruction for large-scale scenes, IEEE TGRS, 62 (2024), p. 3435743
2024
-
[41]
Zhang, J
J. Zhang, J. Chen, Q. W ang, and S. Chen, Spa- tiotemporal saliency detection based on maximum con- sistency superpixels merging for video analysis , IEEE TII, 16 (2019), pp. 606–614
2019
-
[42]
C. Zhao, B. Qin, S. Feng, W. Zhu, W. Sun, W. Li, and X. Jia, Hyperspectral image classification with multi-attention transformer and adaptive super- pixel segmentation-based active learning, IEEE TIP, 32 (2023), pp. 3606–3621
2023
-
[43]
L. Zhu, Q. She, B. Zhang, Y. Lu, Z. Lu, D. Li, and J. Hu, Learning the superpixel in a non-iterative and lifelong manner , in Proceedings of the CVPR, 2021, pp. 1225–1234
2021
-
[44]
D. Zou, H. Peng, X. Huang, R. Yang, J. Li, J. Wu, C. Liu, and P. S. Yu, Se-gsl: A general and effective graph structure learning framework through structural entropy optimization, in Proceedings of the ACM Web Conference 2023, 2023, pp. 499–510. Copyright © 2025 by SIAM Unauth...
2023
-
[600]
As the number of edges decreases, the computation time required for subsequent iterations also decreases accordingly
Note that the algorithm’s time complexity is lin- early related to the number of edges between superpix- els. As the number of edges decreases, the computation time required for subsequent iterations also decreases accordingly. Copyright © 2025 by SIAM Unauthorized reproductio...
2025
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.