REVIEW 4 major objections 6 minor 38 references
TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentation
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper introduces TEMSET-24K, a densely annotated dataset of 24,306 trans-anal endoscopic microsurgery (TEMS) video micro-clips, and claims that its STALNet model can segment surgical timelines with accuracy and F1 up to 0.99 on…
desk verdict Useful new TEMS dataset, but the validation protocol is under-specified and the benchmark numbers are not yet credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
STALNet (Spatiotemporal Adaptive LSTM Network) is the central model: it runs a time-distributed image encoder over each micro-clip, feeds the feature sequence through an adaptive multi-layer LSTM with an attention layer, and outputs three classification heads for phase, task, and action. The complementary machinery is the micro-clip construction, which pairs each labeled keyframe with the 30 to 60 seconds of video leading up to it, giving the model motion context instead of a single still image. The phase-task-action taxonomy itself is the organizing structure that turns a continuous video stream into discrete, searchable timeline labels.
What would settle it
Inspect the validation column in timeline labels.csv: if microclips from the same TEMS surgery appear in both training and validation, recompute accuracy and F1 with a leave-one-surgery-out split; if the scores drop materially, the reported numbers measure memorization of nearby context rather than generalization to new procedures.
Extended reading notes
Core claim
The central claim is that a TEMS procedure can be represented as a three-level timeline of phase, task, and action, and that a spatiotemporal model can predict all three levels directly from short video micro-clips. The paper reports STALNet, tested with ConvNeXt, ViT, and SWIN V2 encoders, achieving accuracy up to 0.99 and F1 up to 0.99 on well-represented phases such as Setup and Suturing, while rare actions like Debris Wash and Haemostatis sit near F1 0.50. The dataset itself, 24,306 micro-clips with keyframes, JSON annotation exports, and a timeline labels table that also records remaining surgical time, is the primary contribution, along with the proposed taxonomy and the preprocessing library that turns raw multi-part videos into machine-learning-ready clips.
Load-bearing premise
The reported accuracy and F1 figures assume that no overlapping microclips from the same surgery are split across the training and validation sets, but the paper does not state how the validation column was constructed.
Editorial extensions
If this is right
- A surgeon or training committee could search a large TEMS video library for specific events, such as bleeding or suturing, without manually reviewing hours of footage.
- Because the dataset includes a time-to-finish column, the same micro-clips can be used to train models that estimate remaining operative time.
- The phase-task-action triplet and the preprocessing recipe transfer to other video-recorded surgical specialties, so the dataset provides a template for similar benchmarks elsewhere.
- Any future model trained on TEMSET-24K can be compared against the reported STALNet numbers, giving the field a common reference point.
Reading between the lines
- Editorial extension: if the validation column splits microclips from the same surgery across training and validation, the headline 0.99 accuracy may overstate generalization; a leave-one-surgery-out evaluation would be the safer estimate of real-world performance.
- Editorial extension: because each label is attached to a 30-to-60-second window ending at a keyframe, the model is likely good at recognizing which stage a moment belongs to, but the dataset does not test precise localization of the boundary frames where one phase becomes another.
- Editorial extension: since access to the dataset is controlled by a restricted application process, reproducibility of the benchmark will depend on approval turnaround; the downloadable code alone does not make the underlying videos public.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces TEMSET-24K, a dataset of microclips extracted from transanal endoscopic microsurgery (TEMS) videos, each annotated by clinical experts with a hierarchical phase/task/action label triplet. The authors also propose STALNet, a spatiotemporal model combining a time-distributed vision encoder (ConvNeXt, ViT, or SWIN V2) with an LSTM and attention, and report high accuracy and F1 scores for phase, task, and action prediction. The paper additionally describes an annotation pipeline, post-processing steps, and a released Python library for video processing.
Significance. If the dataset and its annotations are validated properly, TEMSET-24K would fill a clear gap in surgical data science: a densely annotated, expert-validated dataset for multipart endoscopic video indexing, with a clinically motivated three-level taxonomy. The release of preprocessing code and a benchmark model is a useful contribution to the community. However, the central evidence supporting the dataset's usability—the STALNet benchmark results—currently rests on an undocumented validation split and inconsistent dataset statistics, which must be corrected before the work can be fully credited.
major comments (4)
- [Post-Processing of Annotations to Generate ML-ready Dataset; Model Validation] The construction of the validation set is not specified. The paper only states that 'we added a valid column to indicate the suggested validation set' without describing whether the split is by surgery, by video, or by individual clip. Because Algorithm 1 creates each microclip as a temporal window ending at a keyframe, and keyframes are extracted at approximately 550 per video, microclips from the same surgery share substantial temporal context and, with a 60-second offset, can overlap directly. If the split is at clip level, the reported accuracy and F1 scores (Tables 1-4) may reflect memorization of nearby context rather than generalization to new procedures. This issue is load-bearing for the headline performance claim; the authors must document the split construction and, ideally, report results with a surgery-level split.
- [Abstract; Contributions; Data Records; Technical Validation] There are unreconciled counts across the manuscript. The abstract and contributions state 24,306 microclips and 84 actions, while Data Records states 24,704 microclips and Technical Validation states 21 actions. Additionally, the Discussion refers to '11 tasks' whereas the taxonomy and Table 3 list 12 tasks. These contradictions must be resolved, as they directly affect the dataset integrity and the interpretation of the benchmark results.
- [Technical Validation / Annotation Assessment] No inter-annotator agreement is reported. The paper describes that annotations were initially performed by one surgeon and validated by at least two other surgeons, with conflicts resolved through discussion, but it provides no quantitative agreement metrics such as Cohen's kappa, Fleiss' kappa, or percentage agreement for the phase, task, and action labels. For a dataset whose central contribution is expert annotation, inter-annotator agreement is a standard validation expectation; its absence weakens the claim of consistent, 'densely' annotated labels and makes it difficult to assess label quality.
- [Post-Processing of Annotations; Data Records] The microclip duration is inconsistent. The post-processing section states that microclips capture 'the 30 seconds of video leading up to that frame,' while the Data Records section states that the microclips folder contains '60-second video clips leading up to each keyframe.' This inconsistency directly affects the temporal context of each sample and the leakage analysis. The actual offset used in the released dataset must be clarified.
minor comments (6)
- [Tables 2-4 and Statistical Analysis] Please clarify whether the '±' values are standard deviations or standard errors; the text uses 'variance' but the reported values appear to be standard deviations, and some intervals (e.g., 0.99 ± 0.10) exceed the [0,1] range for accuracy and F1 scores.
- [References] The text says the annotation tool is Label Studio version 1.12.1, but the cited reference [33] is for WebAnno; please provide the correct citation for Label Studio.
- [Data Records] The 'valid column' is mentioned as indicating the suggested validation set, but its contents and usage are not documented; please include an example or a schema of the CSV columns.
- [Figure 6] The ROC curves are presented without AUC values or a clear explanation of how multi-class ROC curves are aggregated; please report AUC scores or specify the averaging method (e.g., one-vs-rest).
- [Equation (2)] The metrics in (2) are defined for binary classification; please clarify how accuracy and F1 are computed for the multiclass phase/task/action predictions, especially the handling of the 'No Action' class and class imbalance.
- [Fig. 5 and Problem Formulation] The term 'Adaptive LSTM' is not clearly explained; the statement that the number of LSTM layers depends on the input sequence length T is unusual and would benefit from a precise definition or rephrasing.
Circularity Check
No significant circularity: the reported benchmark is an empirical evaluation against clinician ground truth, not a derivation that reduces to its inputs.
full rationale
TEMSET-24K is a dataset curation and benchmarking study. Its central outputs are a clinician-defined phase/task/action taxonomy, a set of 24k labeled micro-clips, and measured accuracy/F1 values for STALNet with ConvNeXt, ViT, and SWIN V2 encoders. These are empirical claims. There is no first-principles derivation whose conclusion could be equivalent to an input: the model is trained and evaluated on the same human-annotated labels, which is standard supervised benchmarking rather than a definitional circularity. The encoders and the fastai training library are external, and the clinical context citation (ref. 32, the TREc trial) is an externally published study, not a load-bearing self-citation. The main unresolved issue is that the 'valid column' introduced in Post-Processing is not documented as a surgery-level or patient-level split; because microclips are temporal windows ending at keyframes (Algorithm 1), overlapping clips could leak between training and validation. That is a data-leakage/soundness threat, not a circularity: even if leakage occurred, the reported numbers would be inflated rather than true by construction. No circular step can be exhibited from the paper's equations or citations, so the score is 0.
Assumptions & free parameters
free parameters (4)
- Microclip offset (temporal context length) =
30 s in Methods; 60 s in Data Records
- Keyframe cosine-similarity threshold =
not reported
- STALNet sequence length T =
not reported
- Loss weights alpha, beta, gamma =
not reported
assumptions (5)
- domain assumption Expert annotations exported from Label Studio are treated as ground truth.
- domain assumption Cosine-distance keyframe selection preserves timeline coverage.
- domain assumption A microclip's label equals the label of its terminal keyframe.
- ad hoc to paper Boundary overlaps can be resolved by keeping the trailing label.
- domain assumption Pretrained ImageNet encoders transfer to endoscopic imagery.
Cite this review
Pith. "Pith review of TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentation." pith.science (2026). https://pith.science/paper/37U7GHHB
@misc{pith2026250206708,
author = {Pith},
title = {Pith review of: TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/37U7GHHB}},
note = {Machine review of arXiv:2502.06708}
}
read the original abstract
Indexing endoscopic surgical videos is vital in surgical data science, forming the basis for systematic retrospective analysis and clinical performance evaluation. Despite its significance, current video analytics rely on manual indexing, a time-consuming process. Advances in computer vision, particularly deep learning, offer automation potential, yet progress is limited by the lack of publicly available, densely annotated surgical datasets. To address this, we present TEMSET-24K, an open-source dataset comprising 24,306 trans-anal endoscopic microsurgery (TEMS) video micro-clips. Each clip is meticulously annotated by clinical experts using a novel hierarchical labeling taxonomy encompassing phase, task, and action triplets, capturing intricate surgical workflows. To validate this dataset, we benchmarked deep learning models, including transformer-based architectures. Our in silico evaluation demonstrates high accuracy (up to 0.99) and F1 scores (up to 0.99) for key phases like Setup and Suturing. The STALNet model, tested with ConvNeXt, ViT, and SWIN V2 encoders, consistently segmented well-represented phases. TEMSET-24K provides a critical benchmark, propelling state-of-the-art solutions in surgical data science.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Weiser, T. G. et al. Estimate of the global volume of surgery in 2012: an assessment supporting improved health outcomes. The Lancet 385, S11 (2015)
work page 2015
-
[2]
Nepogodiev, D. et al. Global burden of postoperative death. The Lancet 393, 401 (2019)
work page 2019
-
[3]
England, N. Secure data environment. www.digital.nhs.uk/services/secure-data-environment-service(2021)
work page 2021
-
[4]
NHS, N. Z. Delivering a net zero nhs. https://www.england.nhs.uk/greenernhs/a-net-zero-nhs/ (2020)
work page 2020
-
[5]
Maier-Hein, L. et al. Surgical data science for next-generation interventions. Nat. Biomed. Eng. 1, 691–696 (2017)
work page 2017
-
[6]
Reiley, C. E., Lin, H. C., Yuh, D. D. & Hager, G. D. Review of methods for objective surgical skill evaluation. Surg. endoscopy 25, 356–366 (2011)
work page 2011
-
[7]
Goodman, E. D. et al. A real-time spatiotemporal ai model analyzes skill in open surgical videos. arXiv preprint arXiv:2112.07219 (2021)
arXiv 2021
-
[8]
Machine and deep learning for workflow recognition during surgery
Padoy, N. Machine and deep learning for workflow recognition during surgery. Minim. Invasive Ther. & Allied Technol. 28, 82–90 (2019)
work page 2019
Show all 38 references
-
[9]
Huaulm´e, A. et al. Offline identification of surgical deviations in laparoscopic rectopexy.Artif. Intell. Medicine 104, 101837 (2020)
2020
-
[10]
Kadkhodamohammadi, A. et al. Towards video-based surgical workflow understanding in open orthopaedic surgery. Comput. Methods Biomech. Biomed. Eng. Imaging & Vis. 9, 286–293 (2021)
2021
-
[11]
Holden, M. S. et al. Feasibility of real-time workflow segmentation for tracked needle interventions. IEEE Transactions on Biomed. Eng. 61, 1720–1728 (2014)
2014
-
[12]
Padoy, N. et al. Statistical modeling and recognition of surgical workflow. Med. image analysis 16, 632–641 (2012)
2012
-
[13]
Lin, H. C. et al. Automatic detection and segmentation of robot-assisted surgical motions. In International conference on medical image computing and computer-assisted intervention, 802–810 (Springer, 2005)
2005
-
[14]
& Jannin, P
Dergachyova, O., Bouget, D., Huaulm ´e, A., Morandi, X. & Jannin, P. Automatic data-driven real-time segmentation and recognition of surgical workflow. Int. journal computer assisted radiology surgery 11, 1081–1089 (2016)
2016
-
[15]
Sv-rcnet: workflow recognition from surgical videos using recurrent convolutional network
Jin, Y .et al. Sv-rcnet: workflow recognition from surgical videos using recurrent convolutional network. IEEE transactions on medical imaging 37, 1114–1126 (2017). 16/17
2017
-
[16]
& Navab, N
Blum, T., Feußner, H. & Navab, N. Modeling and segmentation of surgical workflow from laparoscopic video. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2010: 13th International Conference, Beijing, China, September 20-24, 2010, Proceedings, Part III 13, ...
2010
-
[17]
Twinanda, A. P. et al. EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging 36, 86–97 (2016)
2016
-
[18]
Ramesh, S. et al. Multi-task temporal convolutional networks for joint recognition of surgical phases and steps in gastric bypass procedures. Int. journal computer assisted radiology surgery 16, 1111–1119 (2021)
2021
-
[19]
& Heng, P.-A
Gao, X., Jin, Y ., Long, Y ., Dou, Q. & Heng, P.-A. Trans-SVNet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, ...
2021
-
[20]
& Speidel, S
Funke, I., Rivoir, D., Krell, S. & Speidel, S. TUNeS: A Temporal U-Net with Self-Attention for Video-based Surgical Phase Recognition. arXiv preprint arXiv:2307.09997 (2023)
2023 arXiv
-
[21]
Video-based content analysis
Huber, M. Video-based content analysis. In Campbell, A. G., Hong, L., Meinel, F. & Zallio, M. (eds.)Handbook of Research on Multimodal Human Computer Interaction and Pervasive Services, 10.4324/9780429316647-5 (CRC Press, 2020)
-
[22]
& ¨Ozsu, M
Liu, L. & ¨Ozsu, M. T. Encyclopedia of database systems, vol. 6 (Springer New York, 2009)
2009
-
[23]
Feldman, L. S. et al. SAGES Video-Based Assessment (VBA) program: a vision for life-long learning for surgeons. Surg. endoscopy 34, 3285–3288 (2020)
2020
-
[24]
& Navab, N
Vercauteren, T., Unberath, M., Padoy, N. & Navab, N. CAI4CAI: the rise of contextual artificial intelligence in computer-assisted interventions. Proc. IEEE 108, 198–214 (2019)
2019
-
[25]
Nwoye, C. I. et al. Recognition of instrument-tissue interactions in endoscopic videos via action triplets. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23, 364–37...
2020
-
[26]
Mascagni, P. et al. Computer vision in surgery: from potential to clinical value. npj Digit. Medicine 5, 163 (2022)
2022
-
[27]
Lewandrowski, K.-U. et al. Regional variations in acceptance, and utilization of minimally invasive spinal surgery techniques among spine surgeons: results of a global survey. J. Spine Surg. 6, S260 (2020)
2020
-
[28]
Richards, M. K. et al. A national review of the frequency of minimally invasive surgery among general surgery residents: assessment of ACGME case logs during 2 decades of general surgery resident training. JAMA surgery 150, 169–172 (2015)
2015
-
[29]
& Buhmann, J
Paysan, D., Haug, L., Bajka, M., Oelhafen, M. & Buhmann, J. M. Self-supervised representation learning for surgical activity recognition. Int. J. Comput. Assist. Radiol. Surg. 16, 2037–2044 (2021)
2021
-
[30]
Valderrama, N. et al. Towards holistic surgical scene understanding. In International conference on medical image computing and computer-assisted intervention, 442–452 (Springer, 2022)
2022
-
[31]
Ayobi, N. et al. Pixel-wise recognition for holistic surgical scene understanding. arXiv preprint arXiv:2401.11174 (2024)
2024 arXiv
-
[32]
Bach, S. P. et al. Radical surgery versus organ preservation via short-course radiotherapy followed by transanal endoscopic microsurgery for early-stage rectal cancer (trec): a randomised, open-label feasibility study. The Lancet Gastroenterol. & Hepatol. 6, 92–105 (2021)
2021
-
[33]
Gurevych, I., De Castilho, R. E. & Biemann, C. Webanno: A flexible, web-based and visually supported system for distributed annotations. 51st Annu. Meet. (2013)
2013
-
[34]
Liu, Z. et al. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11976–11986 (2022)
2022
-
[35]
Liu, Z. et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12009–12019 (2022)
2022
-
[36]
Steiner, A. et al. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270 (2021)
2021 arXiv
-
[37]
Caron, M. et al. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, 9650–9660 (2021)
2021
-
[38]
& Gugger, S
Howard, J. & Gugger, S. Fastai: a layered api for deep learning. Information 11, 108 (2020). 17/17
2020
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.