REVIEW 3 major objections 5 minor 28 references
Hierarchical Spatio-temporal Segmentation Network for Ejection Fraction Estimation in Echocardiography Videos
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read HSS-Net: a hierarchical CNN-Mamba video segmentation network that estimates ejection fraction from echocardiography videos more accurately than eight baselines on three datasets.
desk verdict Solid segmentation paper with a real protocol flaw: EF numbers come from oracle-cropped clips (ED first, ES last), so the headline 'EF estimation from videos' overstates what is actually evaluated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the hierarchical encoder-decoder combined with the STCS scanning module. Low-level stages use separable convolution blocks (MobileNetV2-style inverted residuals) on single frames to retain fine local detail. High-level stages use spatio-temporal Mamba blocks that reshape the video into a long one-dimensional sequence and pass it through STCS, which scans along bidirectional temporal, spatial, diagonal, and anti-diagonal routes. This lets a selective state-space model build long-range cross-frame and cross-position dependencies, capturing structures such as apex motion and lateral wall contraction that a single-frame segmentation would miss, while avoiding the full co
What would settle it
Retrain and test HSS-Net on full-length, unaligned echo clips without physician-provided ED/ES alignment, then compare the EF Pearson correlation with the reported aligned numbers (90.47 on CAMUS, 84.50 on EchoNet-Dynamic). If the correlation falls toward the image-only baseline or the bias grows beyond the reported values, the advantage is partly an artifact of the alignment protocol; testing on randomly cropped non-aligned clips would also reveal timing leakage.
Extended reading notes
Core claim
The paper's central claim is that deliberately separating single-frame detail from multi-frame motion modeling improves ejection-fraction estimation. The proposed HSS-Net keeps convolutional blocks in the low-level encoder stages to preserve edges, textures, and boundary detail, and switches to spatio-temporal Mamba blocks in the high-level stages to capture how the ventricle moves across frames. To keep the multi-frame modeling from staying local, the Spatio-temporal Cross Scan (STCS) module flattens video patches into one-dimensional sequences and scans them along temporal, spatial, diagonal, and anti-diagonal directions in forward and backward passes. On CAMUS, HSS-Net reports a Pearson c
Load-bearing premise
The load-bearing premise is that each input clip is pre-aligned so the end-diastolic frame is the first frame and the end-systolic frame is the last frame, using physician-provided timing; real workflows must discover those two frames from an unaligned video, and the reported EF correlations may depend on having been given that timing.
Editorial extensions
If this is right
- Low-level single-frame detail and high-level video modeling are complementary: the paper's ablations show that removing either (image-only or video-only variants) lowers EF correlation on both CAMUS and EchoNet-Dynamic.
- Every STCS scan direction contributes: dropping temporal, spatial, diagonal, or anti-diagonal scanning degrades at least one EF metric, which supports the design of multi-perspective scanning.
- The model's per-frame cost (5.6 GFLOPs, 31.2M parameters) is in the range of much weaker baselines, suggesting the accuracy gain does not require a large computational budget.
- On the two larger datasets the reported EF bias is small (1.29 ± 8.68 on EchoNet-Pediatric, 0.95 ± 6.75 on EchoNet-Dynamic), which is the clinically relevant quantity for screening and follow-up.
Reading between the lines
- The reported numbers depend on a protocol where each clip is cropped so the physician-annotated end-diastolic frame is first and the end-systolic frame is last; whether the advantage survives without that timing alignment is not tested in the paper and is the most direct open question.
- Because STCS's diagonal scanning is a generic mechanism, the same hierarchical CNN-Mamba recipe may transfer to other sparsely annotated video segmentation tasks, such as fetal ultrasound or cardiac MRI, where only two key frames are labelled.
- The paper still follows the segmentation-then-EF pipeline; a direct end-to-end regression head on the same spatio-temporal features could potentially capture motion cues that the two-frame volume formula discards.
- The reported bias and standard deviation on CAMUS are close to the range of inter-observer variability typically reported in echocardiography, so part of the remaining error may be label noise rather than architecture; an observer-agreement comparison would clarify the practical ceiling.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HSS-Net, a hierarchical spatio-temporal segmentation network for echocardiography videos, combining low-level separable convolution blocks (single-frame detail) with high-level spatio-temporal Mamba blocks (multi-frame global dynamics). A Spatio-temporal Cross Scan (STCS) module with temporal, spatial, diagonal, and anti-diagonal scanning modes is introduced to capture long-range dependencies. EF is computed from predicted ED/ES contours using the Simpson method. The authors report state-of-the-art results on CAMUS, EchoNet-Pediatric, and EchoNet-Dynamic, with correlations of 90.47, 76.91, and 84.50 respectively, along with Dice and HD95 improvements over several segmentation baselines. The code is publicly available.
Significance. If the results hold, HSS-Net offers an efficient (5.6G FLOPs, 31.2M params) hybrid architecture that balances local detail and global temporal modeling, with a credible conceptual motivation for reducing EF bias caused by boundary segmentation errors. The paper ships code and follows a reproducible evaluation protocol, which are strengths. However, the headline EF-estimation claim is weakened by the evaluation protocol: clips are truncated so that the ground-truth ED frame is first and ES frame is last, so the model never discovers keyframe timing. The reported correlations are thus conditional on oracle timing, and the advantage over segmentation baselines is shown on this aligned protocol only. The lack of confidence intervals or significance tests further limits the strength of the SOTA claim.
major comments (3)
- [Section 3, Datasets] The evaluation protocol crops every input clip so that the ground-truth ED frame is the first frame and the ground-truth ES frame is the last frame, and only these two frames are supervised and used for EF computation. This oracle timing leaks the keyframe identities that a real EF estimator must discover from unaligned video. The reported correlations (90.47/76.91/84.50) therefore measure segmentation quality on physician-timed keyframes, not end-to-end EF estimation. The comparison to baselines is internally fair, but the absolute SOTA claim does not transfer to unaligned clinical videos. Please add an experiment on unaligned clips (e.g., with a separate ED/ES detector or by evaluating on full-length videos) or explicitly re-frame the claim as keyframe-conditional segmentation.
- [Tables 1-2, Section 3.1] All reported numbers are single-run point estimates without confidence intervals, repeated-seed variance, or significance tests. Several Dice gaps are small (e.g., 93.89 vs 93.02 on CAMUS; 92.67 vs 92.48 on EchoNet-Dynamic), so the claimed advantage over baselines may not be statistically reliable. Please report mean±std over at least three runs and apply paired tests (e.g., Wilcoxon or permutation) to the EF correlation and Dice differences.
- [Section 3.1, Tables 1-2] The SOTA claim for EF estimation is supported only by comparisons against segmentation baselines (UNet++, TransUNet, VideoMamba, etc.). No direct EF regression or keyframe-detection method is compared, although these methods are the relevant SOTA for EF estimation from videos (e.g., EchoNet). Please include comparisons to such methods under the same aligned protocol and, if possible, on unaligned inputs, or restrict the claim to segmentation-based EF estimation on aligned keyframes.
minor comments (5)
- [Table 2] The entry '84.500.95±6.75' in the EchoNet-Dynamic row is missing a space; it should read '84.50 0.95±6.75'.
- [Section 1] The sentence 'The primary of echocardiography analysis' is ungrammatical; it should be 'The primary goal of echocardiography analysis' or similar.
- [Section 3, Datasets] The text says CAMUS has 'annotations provided for all frames' but later states that only ED and ES annotations are used for training and evaluation. Please clarify which annotations are actually used.
- [Throughout] The term 'spatio-temporal' is used with inconsistent hyphenation ('spatio-temporal' vs 'spatiotemporal'). Please unify the notation.
- [Equations (1)-(2)] Using the same symbol F_i for both input and output of the residual blocks is acceptable but could be clearer with an explicit out-notation (e.g., F_i^out).
Circularity Check
No significant circularity: the EF values are computed from predicted segmentation via the standard Simpson method, and the architecture's empirical claims are tested on external benchmarks.
full rationale
The paper's central claim is that HSS-Net improves ejection-fraction estimation by improving left-ventricular endocardium segmentation. The derivation chain is not circular: the model is trained on ground-truth segmentation masks of ED and ES frames (with Dice and cross-entropy losses), not on EF labels. EF is then computed deterministically from the predicted contours using Simpson's method, as stated in Section 3: 'ground truth and predicted ejection fractions are obtained using the Simpson's single-plane method of disks' (EchoNet) and 'the Simpson's biplane method of disks' (CAMUS). No equation defines the output in terms of itself, and no fitted parameter is later renamed as a prediction. The architectural components are drawn from cited prior work (Mamba, MobileNetV2, etc.), but these citations are not load-bearing in the sense of substituting for empirical validation; the reported correlations, biases, Dice, and HD95 are measured against external benchmark datasets with fixed splits. The authors' own prior papers ([18], [19]) are cited only for general ultrasound-segmentation context, not to justify the uniqueness or validity of the proposed method. The one noteworthy validity concern is the evaluation protocol described in Section 3: 'The video clips are cropped to ensure that the ED frame is the first and the ES frame is the last.' This means the model never has to locate ED/ES frames in unaltered videos, so the absolute SOTA numbers may overstate performance in a fully end-to-end clinical workflow where keyframe timing must be discovered. However, this is an external-validity limitation, not a circularity: the same aligned protocol is applied to all compared methods, the EF labels are not used as training targets, and no step of the derivation reduces to its own input. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Loss balance weight alpha =
0.8
- Number of sampled frames T =
10
- ED-first/ES-last clip construction =
ED first, ES last
assumptions (4)
- domain assumption Simpson's method of disks converts endocardial contours to volumes and EF.
- domain assumption Mamba S6 selective state-space models capture long-range spatio-temporal dependencies in video.
- domain assumption Ground-truth ED/ES masks are accurate and correspond to true endocardial boundaries.
- domain assumption Uniformly sampled 10-frame clips from ED to ES represent a complete heartbeat cycle.
Cite this review
Pith. "Pith review of Hierarchical Spatio-temporal Segmentation Network for Ejection Fraction Estimation in Echocardiography Videos." pith.science (2026). https://pith.science/paper/G3EV6W3O
@misc{pith2026250818681,
author = {Pith},
title = {Pith review of: Hierarchical Spatio-temporal Segmentation Network for Ejection Fraction Estimation in Echocardiography Videos},
year = {2026},
howpublished = {\url{https://pith.science/paper/G3EV6W3O}},
note = {Machine review of arXiv:2508.18681}
}
read the original abstract
Automated segmentation of the left ventricular endocardium in echocardiography videos is a key research area in cardiology. It aims to provide accurate assessment of cardiac structure and function through Ejection Fraction (EF) estimation. Although existing studies have achieved good segmentation performance, their results do not perform well in EF estimation. In this paper, we propose a Hierarchical Spatio-temporal Segmentation Network (\ourmodel) for echocardiography video, aiming to improve EF estimation accuracy by synergizing local detail modeling with global dynamic perception. The network employs a hierarchical design, with low-level stages using convolutional networks to process single-frame images and preserve details, while high-level stages utilize the Mamba architecture to capture spatio-temporal relationships. The hierarchical design balances single-frame and multi-frame processing, avoiding issues such as local error accumulation when relying solely on single frames or neglecting details when using only multi-frame data. To overcome local spatio-temporal limitations, we propose the Spatio-temporal Cross Scan (STCS) module, which integrates long-range context through skip scanning across frames and positions. This approach helps mitigate EF calculation biases caused by ultrasound image noise and other factors.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Journal of Clinical Medicine10(7), 1391 (2021)
Akkus, Z., Aly, Y.H., Attia, I.Z., Lopez-Jimenez, F., Arruda-Olson, A.M., Pellikka, P.A., Pislaru, S.V., Kane, G.C., Friedman, P.A., Oh, J.K.: Artificial intelligence (ai)-empowered echocardiography interpretation: a state-of-the-art review. Journal of Clinical Medicine10(7), 1391 (2021)
work page 2021
-
[2]
In: International Conference on Computer Vision
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: Vivit: A video vision transformer. In: International Conference on Computer Vision. pp. 6836–6846 (2021)
2021
-
[3]
Frontiers in Cardiovascular Medicine 7, 25 (2020)
Chen,C.,Qin,C.,Qiu,H.,Tarroni,G.,Duan,J.,Bai,W.,Rueckert,D.:Deeplearn- ing for cardiac image segmentation: a review. Frontiers in Cardiovascular Medicine 7, 25 (2020)
work page 2020
-
[4]
arXiv preprint arXiv:2102.04306 (2021)
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., Zhou, Y.:Transunet:Transformersmakestrongencodersformedicalimagesegmentation. arXiv preprint arXiv:2102.04306 (2021)
arXiv 2021
-
[5]
In: Computer Vision and Pattern Recogni- tion
Deng, X., Wu, H., Zeng, R., Qin, J.: Memsam: Taming segment anything model for echocardiography video segmentation. In: Computer Vision and Pattern Recogni- tion. pp. 9622–9631 (2024)
work page 2024
-
[6]
arXiv preprint arXiv:2312.00752 (2023)
Gu, A., Dao, T.: Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023)
arXiv 2023
-
[7]
IEEE Transactions on Medical Imaging 42(9), 2763–2775 (2023)
He, A., Wang, K., Li, T., Du, C., Xia, S., Fu, H.: H2former: An efficient hierarchical hybrid transformer for medical image segmentation. IEEE Transactions on Medical Imaging 42(9), 2763–2775 (2023)
work page 2023
-
[8]
IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control67(12), 2519–2530 (2020)
Leclerc, S., Smistad, E., Østvik, A., Cervenansky, F., Espinosa, F., Espeland, T., Berg, E.A.R., Belhamissi, M., Israilov, S., Grenier, T., et al.: Lu-net: a multi- stage attention network to improve the robustness of segmentation of left ven- tricular structures in 2-d echocardiography. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Cont...
work page 2020
Show all 28 references
-
[9]
IEEE Transactions on Medical Imaging38(9), 2198–2210 (2019)
Leclerc, S., Smistad, E., Pedrosa, J., Ostvik, A., Cervenansky, F., Espinosa, F., Es- peland,T.,Berg,E.A.R.,Jodoin,P.M.,Grenier,T.,Lartizien,C.,Dhooge,J.,Lovs- takken, L., Bernard, O.: Deep learning for segmentation using an open large-scale dataset in 2d echocardiography. IEE...
2019
-
[10]
arXiv preprint arXiv:2403.06977 (2024)
Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., Qiao, Y.: Videomamba: State space model for efficient video understanding. arXiv preprint arXiv:2403.06977 (2024)
2024 arXiv
-
[11]
In: Fourth International Conference on 3D Vision
Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: Fourth International Conference on 3D Vision. pp. 565–571 (2016) 10 D. Wang et al
2016
-
[12]
Physica Medica67, 58–69 (2019)
Moradi, S., Oghli, M.G., Alizadehasl, A., Shiri, I., Oveisi, N., Oveisi, M., Maleki, M., Dhooge, J.: Mfp-unet: A novel deep learning based approach for left ventricle segmentation in echocardiography. Physica Medica67, 58–69 (2019)
2019
-
[13]
IEEE Transactions on Medical Imaging 41(10), 2867–2878 (2022)
Painchaud, N., Duchateau, N., Bernard, O., Jodoin, P.M.: Echocardiography seg- mentation with enforced temporal consistency. IEEE Transactions on Medical Imaging 41(10), 2867–2878 (2022)
2022
-
[14]
Journal of the American Society of Echocardiography36(5), 482–489 (2023)
Reddy, C.D., Lopez, L., Ouyang, D., Zou, J.Y., He, B.: Video-based deep learning for automated assessment of left ventricular ejection fraction in pediatric patients. Journal of the American Society of Echocardiography36(5), 482–489 (2023)
2023
-
[15]
In: Computer Vision and Pattern Recog- nition
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: In- verted residuals and linear bottlenecks. In: Computer Vision and Pattern Recog- nition. pp. 4510–4520 (2018)
2018
-
[16]
In: Medical Image Computing and Computer Assisted Intervention
Thomas, S., Gilbert, A., Ben-Yosef, G.: Light-weight spatio-temporal graphs for segmentation and ejection fraction prediction in cardiac ultrasound. In: Medical Image Computing and Computer Assisted Intervention. vol. 13434, pp. 380–390 (2022)
2022
-
[17]
Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. vol. 30 (2017)
2017
-
[18]
In: International Symposium on Biomedical Imaging
Wang,D.,Zhou,T.,Yang,J.:Hybrid-frequencyfeatureevolutionnetworkforendo- scopic ultrasound image segmentation. In: International Symposium on Biomedical Imaging. pp. 1–5 (2025)
2025
-
[19]
IEEE Transactions on Circuits and Systems for Video Technology pp
Wang, D., Zhou, T., Zhang, Y., Gao, S., Yang, J.: Frequency-aware interaction network for ultrasound image segmentation. IEEE Transactions on Circuits and Systems for Video Technology pp. 1–1 (2024)
2024
-
[20]
Medical Image Analysis 84, 102686 (2023)
Wei, H., Ma, J., Zhou, Y., Xue, W., Ni, D.: Co-learning of appearance and shape for precise ejection fraction estimation from echocardiographic sequences. Medical Image Analysis 84, 102686 (2023)
2023
-
[21]
In: Proceedings of the AAAI Conference on Artificial Intelligence
Wu, H., Lin, J., Xie, W., Qin, J.: Super-efficient echocardiography video segmen- tation via proxy-and kernel-based semi-supervised learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 2803–2811 (2023)
2023
-
[22]
Medical Image Analysis78, 102397 (2022)
Wu,H.,Liu,J.,Xiao,F.,Wen,Z.,Cheng,L.,Qin,J.:Semi-supervisedsegmentation of echocardiography videos via noise-resilient spatiotemporal semantic calibration and fusion. Medical Image Analysis78, 102397 (2022)
2022
-
[23]
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. vol. 34, pp. 12077–12090 (2021)
2021
-
[24]
In: International Conference on Computer Vision
Yang, J., Ding, X., Zheng, Z., Xu, X., Li, X.: Graphecho: Graph-driven unsuper- vised domain adaptation for echocardiogram video segmentation. In: International Conference on Computer Vision. pp. 11844–11853 (2023)
2023
-
[25]
arXiv preprint arXiv:2401.14168 (2024)
Yang, Y., Xing, Z., Zhu, L.: Vivim: a video vision mamba for medical video object segmentation. arXiv preprint arXiv:2401.14168 (2024)
2024 arXiv
-
[26]
arXiv preprint arXiv:2402.08506 (2024)
Ye, Z., Chen, T.: P-mamba: Marrying perona malik diffusion with mamba for efficient pediatric echocardiographic left ventricular segmentation. arXiv preprint arXiv:2402.08506 (2024)
2024 arXiv
-
[27]
Medical Image Analysis 80, 102438 (2022)
Zamzmi, G., Rajaraman, S., Hsu, L.Y., Sachdev, V., Antani, S.: Real-time echocar- diography image analysis and quantification of cardiac indices. Medical Image Analysis 80, 102438 (2022)
2022
-
[28]
IEEE Transac- tions on Medical Imaging39(6), 1856–1867 (2019)
Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N., Liang, J.: Unet++: Redesigning skip connections to exploit multiscale features in image segmentation. IEEE Transac- tions on Medical Imaging39(6), 1856–1867 (2019)
2019
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.