REVIEW 5 major objections 5 minor 1 cited by
Machine Learning-Based Automated Assessment of Intracorporeal Suturing in Laparoscopic Fundoplication
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read An automated video-analysis pipeline tracks both laparoscopic instruments, extracts their motion, and classifies the surgeon as novice or expert with up to 0.817 accuracy and an F1 of 0.806, using no human-annotated surgical videos.
desk verdict Useful integrated pipeline and new dataset, but the headline accuracy and 'no human annotation' claim are not yet supported: likely CV leakage and circular tracking validation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the automated annotation and tracking chain: YOLOv8 produces approximate bounding boxes for surgical tools, the Segment Anything Model converts those boxes into precise segmentation masks, and a retrained detection model tracks the tooltips through the video; a low-pass Butterworth filter at 24 Hz smooths the trajectories. The classification then proceeds through two streams: a supervised stream that computes RMS velocity, RMS acceleration, RMS jerk, total path length, and bimanual dexterity, and an unsupervised stream that renders x-y tooltip trajectories as line-plot images, trains a denoising autoencoder, and classifies the resulting 128-dimensional encodings with a 1-D CNN.
What would settle it
Re-run the skill classification with a leave-one-video-out or leave-one-surgeon-out cross-validation; if accuracy falls substantially below the reported 0.817, the human-annotation-free claim only holds for seen videos.
Extended reading notes
Core claim
The central claim is that a fully automated video-analysis pipeline can grade laparoscopic suturing skill at a level competitive with human annotation, using no manually labeled surgical videos. The pipeline combines a detector that proposes tool bounding boxes with a segmentation model that refines them into precise instrument masks; it then tracks tooltips, filters the trajectories at 24 Hz, and feeds the resulting motion into either handcrafted kinematic features or learned representations. On 141 suturing segments from 28 surgeons performing Nissen fundoplication on porcine bowel, the best result is the unsupervised 1-D CNN with accuracy 0.817 ± 0.108 and F1 0.806 ± 0.110, which the paper presents as superior to supervised feature-based classification with 0.795 accuracy.
Load-bearing premise
The load-bearing assumption is that the 141 suturing segments can be treated as independent samples even though many come from the same video and the same surgeon; if folds mix segments from one video, the reported accuracy could reflect memorization rather than general skill.
Editorial extensions
If this is right
- Trainees could get immediate, automatic skill labels after each suturing segment, removing the bottleneck of expert video review.
- The unsupervised DAE route shows that hand-crafted kinematic metrics are not necessary, simplifying deployment to new tasks.
- Because the motion is extracted from ordinary 2-D laparoscopic video, the approach avoids external sensors and robotic-system kinematics.
- With additional fine-tuning of the tool detector, the pipeline could transfer to other laparoscopic procedures where instrument types differ.
Reading between the lines
- A direct consequence the paper does not test: segment-level cross-validation may overstate generalizability; a leave-one-video-out split would reveal whether accuracy survives seeing a surgeon's entire video only at test time.
- Extending the pipeline with monocular depth estimation, which the paper flags as future work, could separate true instrument motion from camera motion and improve the skill signal.
- The DAE's learned 128-dimensional encodings might recapitulate the handcrafted kinematic features; comparing them directly would clarify what the unsupervised model actually learns.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript describes a fully automated pipeline for assessing laparoscopic suturing skill from videos of Nissen fundoplication on a porcine model. Tool tracking is obtained by combining a YOLOv8 detector with the Segment Anything Model (SAM) to generate segmentations without manual annotation, followed by kinematic feature extraction and classification into novice versus expert. The authors compare supervised classifiers on engineered kinematic features with an unsupervised denoising autoencoder (DAE) whose learned representations are classified by a 1-D CNN and other models. They report a best supervised accuracy of 0.795 and a best DAE-based accuracy of 0.817, concluding that the system can classify performance independent of human annotation of surgical videos.
Significance. If the results hold, the pipeline would be a useful step toward scalable, low-cost surgical skill assessment, and the integration of SAM with YOLOv8 to generate training labels is a constructive idea that could reduce annotation burden. The comparison between hand-crafted kinematic features and DAE-learned representations is also informative. However, the two central validation claims are currently not established: the tool-tracking accuracy is evaluated against masks generated by the same pipeline rather than human annotations, and the skill-classification accuracy is likely inflated by using suture segments from the same subjects or videos in both training and test folds. The paper is therefore promising but requires substantial re-analysis before the headline conclusion can be accepted.
major comments (5)
- [Section 2.4.1] The 10-fold cross-validation is applied to 141 samples that are not independent. Section 2.1 reports 49 videos from 28 subjects, and Section 2.3 states that each video contributes up to three suture segments (S1, S2, S3) that are used as separate samples. Because the split is described only at the sample level, with no grouping by subject or by video, segments from the same video or surgeon can appear in both training and test folds. This allows the classifier to memorize video- or surgeon-specific artifacts rather than learn general skill, inflating the reported accuracy and F1 scores. The effective number of independent units is 28 subjects, far smaller than 141. Please rerun the evaluation with group-based cross-validation, such as leave-one-subject-out or leave-one-video-out, and report performance with confidence intervals based on the number of subjects.
- [Section 3.1.2] The tracking accuracy claim of mAP 0.95 on the fundoplication data is circular. Section 2.2.3 says that 1401 frames were selected because they 'displayed high-quality segmentation with accurately generated masks,' and no human-annotated test set is described. The masks used as ground truth are therefore produced by the same YOLOv8-plus-SAM pipeline that is being validated. This does not support the claim that manual annotation has been eliminated. Please evaluate the tracking model on a held-out set of frames annotated by humans, describe the annotation protocol, and report inter-annotator agreement.
- [Abstract and Table 2] The abstract states that the unsupervised 1-D CNN achieved superior results with an accuracy of 0.817 and an F1 score of 0.806, but Table 2 shows that Random Forest achieved a higher accuracy (0.822), higher positive predictive value (0.845), and higher negative predictive value (0.887) when used with DAE features. The 1-D CNN is better only on F1 score. Please specify the primary outcome used for model selection and qualify the 'superior' claim accordingly, or the abstract should be revised to reflect the metric-dependent nature of the comparison.
- [Sections 2.4.1 and 2.4.2] The manuscript reports 141 samples for the supervised machine-learning classification but 196 samples for the DAE pipeline. The relationship between these two sample sets is not explained, making it impossible to determine whether the two approaches are evaluated on the same cohort and whether the comparison in Tables 1 and 2 is fair. Please clarify how the 196 DAE samples are constructed (e.g., concatenation of S1-S3 for each tool and axis) and reconcile the counts with the 141-sample dataset.
- [Sections 2.1 and 2.3] The exclusion of 11 videos due to incomplete tasks or camera instability, and the exclusion of S4 segments because they were performed by only 2-3 subjects, is reported without checking whether these exclusions are balanced across the novice and expert groups. If excluded videos are disproportionately from one skill group, the reported accuracies may reflect a biased sample. Please report exclusions by group and, if feasible, run a sensitivity analysis including the excluded videos or segments.
minor comments (5)
- [Abstract] The term 'unsupervised' is applied to the 1-D CNN, but the final classification uses PGY-level labels; only the DAE feature-learning stage is unsupervised. Please revise the wording to avoid implying that skill labels are not used.
- [Section 2.4.2] The denoising autoencoder is cited as [25,26], but reference [25] is the COMPASS paper and reference [26] is Desir et al.; neither appears to describe DAE. Please check and correct the citations.
- [Tables 1 and 2] The plus/minus values are not defined. Please state whether they are standard deviations across the 10 folds and report 95% confidence intervals given the small number of subjects.
- [Section 2.2.3] The sentence 'Only 1401 frames that displayed high-quality segmentation with accurately generated masks were selected' introduces a selection step that may bias the training data. Please describe how 'high-quality' was determined and whether the selection was automated or involved human review.
- [Figure 5] The caption says 'Raw Data versus Filtered Data,' but the axes are not labeled. Please label the x- and y-axes and state what the plotted traces represent.
Circularity Check
Tool-tracking validation is self-referential: the reported mAP 0.95 is computed against SAM-generated masks rather than independent human ground truth; the final skill classification uses independent PGY labels, so circularity is partial.
-
self definitional
[Sections 2.2.2, 2.2.3, and 3.1.2]
"The extracted frames (input images) along with bounding boxes were then passed to SAM to generate masks. ... In total, only 1401 frames that displayed high-quality segmentation with accurately generated masks were selected. ... From the dataset of 1401 images, the combination of the YOLOv8 and SAM increased the mAP from 0.76 to 0.95 for the mask of the grasper class."
The mAP 'ground truth' is the output of the very SAM-based automatic annotation pipeline being validated. Section 2.2.2 creates masks by passing frames and YOLOv8 boxes to SAM; Section 2.2.3 uses those auto-generated masks to train, validate, and test the retrained YOLOv8, selecting 1401 frames described only as having 'accurately generated masks.' No human-annotated test set is described. The 0.95 mAP therefore measures agreement between YOLOv8 and SAM, not segmentation accuracy against an external reference. The claim that the model 'accurately detected and segmented the grasper tool' is thus self-referential: the prediction target is defined by the system under test. The skill-classification labels (PGY level) are independent, which is why the overall circularity is only partial.
full rationale
The paper's central skill-classification result is not circular by construction: novice/expert labels come from independent PGY-level groupings, and the kinematic features are computed from tracked tool paths, not from the labels. The 10-fold cross-validation, however, treats 141 samples derived from 49 videos of 28 subjects as independent units without subject- or video-level grouping (Sections 2.1, 2.3, 2.4.1), which risks leakage and inflated accuracy; this is a serious methodological concern but is not a definitional circularity. The one clear circular step is the tool-tracking validation: the mAP 0.95 is evaluated against SAM-generated masks produced by the same automated annotation pipeline that the paper is proposing, so the tracking accuracy claim reduces to self-comparison rather than external verification. The abstract also overstates independence from human annotation because the beginning and end of each suturing step appear to be manually segmented (Methods), although this is an overclaim issue rather than a circularity. No load-bearing self-citation chain or imported uniqueness theorem was found; reference [26] supplies participant demographics and prior validation context but does not force the present results. Overall, one component prediction—the tracking accuracy—reduces to its own input by construction, while the final classification retains independent content, giving a partial-circularity score of 6.
Assumptions & free parameters
free parameters (6)
- Low-pass Butterworth filter cutoff =
24 Hz
- PCA number of components =
3
- DAE latent dimension =
128
- DAE Gaussian noise factor =
0.5
- Frames sampled per video for YOLOv8 training =
200
- DAE input image size =
224 x 224 x 3
assumptions (5)
- domain assumption Kinematic metrics (RMS velocity, RMS acceleration, RMS jerk, path length, bimanual dexterity) are reliable predictors of laparoscopic surgical skill.
- domain assumption Grouping PGY3-5, fellows, and attendings as experts and PGY1-2 as novices yields a meaningful binary skill distinction.
- ad hoc to paper SAM-generated masks are accurate enough to serve as ground truth for training and testing the tool tracking model.
- domain assumption The 24 Hz low-pass filter removes noise without removing meaningful motion information.
- ad hoc to paper Excluding 11 videos (incomplete tasks, camera instability) and segment S4 does not bias the sample.
Cite this review
Pith. "Pith review of Machine Learning-Based Automated Assessment of Intracorporeal Suturing in Laparoscopic Fundoplication." pith.science (2026). https://pith.science/paper/ZYYWESVK
@misc{pith2026241216195,
author = {Pith},
title = {Pith review of: Machine Learning-Based Automated Assessment of Intracorporeal Suturing in Laparoscopic Fundoplication},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZYYWESVK}},
note = {Machine review of arXiv:2412.16195}
}
read the original abstract
Automated assessment of surgical skills using artificial intelligence (AI) provides trainees with instantaneous feedback. After bimanual tool motions are captured, derived kinematic metrics are reliable predictors of performance in laparoscopic tasks. Implementing automated tool tracking requires time-intensive human annotation. We developed AI-based tool tracking using the Segment Anything Model (SAM) to eliminate the need for human annotators. Here, we describe a study evaluating the usefulness of our tool tracking model in automated assessment during a laparoscopic suturing task in the fundoplication procedure. An automated tool tracking model was applied to recorded videos of Nissen fundoplication on porcine bowel. Surgeons were grouped as novices (PGY1-2) and experts (PGY3-5, attendings). The beginning and end of each suturing step were segmented, and motions of the left and right tools were extracted. A low-pass filter with a 24 Hz cut-off frequency removed noise. Performance was assessed using supervised and unsupervised models, and an ablation study compared results. Kinematic features--RMS velocity, RMS acceleration, RMS jerk, total path length, and Bimanual Dexterity--were extracted and analyzed using Logistic Regression, Random Forest, Support Vector Classifier, and XGBoost. PCA was performed for feature reduction. For unsupervised learning, a Denoising Autoencoder (DAE) model with classifiers, such as a 1-D CNN and traditional models, was trained. Data were extracted for 28 participants (9 novices, 19 experts). Supervised learning with PCA and Random Forest achieved an accuracy of 0.795 and an F1 score of 0.778. The unsupervised 1-D CNN achieved superior results with an accuracy of 0.817 and an F1 score of 0.806, eliminating the need for kinematic feature computation. We demonstrated an AI model capable of automated performance classification, independent of human annotation.
Figures
Forward citations
Cited by 1 Pith paper
-
Efficient Frame Extraction: A Novel Approach Through Frame Similarity and Surgical Tool Tracking for Video Segmentation
Kinematics Adaptive Frame Recognition selects frames with significant tool motion, reducing training data 5x to 10x and modestly improving phase segmentation accuracy over uniform sampling.
Reference graph
Works this paper leans on
-
[1]
Birkmeyer, J.D., et al., Surgical Skill and Complication Rates after Bariatric Surgery, in New England Journal of Medicine. 2013. p. 1434-1442
work page 2013
-
[2]
2016, Lippincott Williams and Wilkins
Hogg, M.E., et al., Grading of surgeon technical performance predicts postoperative pancreatic fistula for pancreaticoduodenectomy independent of patient-related variables, in Annals of Surgery. 2016, Lippincott Williams and Wilkins. p. 482-489
work page 2016
-
[3]
2020, American Medical Association
Stulberg, J.J., et al., Association between Surgeon Technical Skills and Patient Outcomes, in JAMA Surgery. 2020, American Medical Association. p. 960-968
work page 2020
-
[4]
IEEE Robotics and Automation Letters, 2023
Anastasiou, D., et al., Keep your eye on the best: contrastive regression transformer for skill assessment in robotic surgery. IEEE Robotics and Automation Letters, 2023. 8(3): p. 1755-1762
work page 2023
-
[5]
Pugh, C.M., D.A. Hashimoto, and J.R. Korndorffer, The what? How? And Who? Of video based assessment, in American Journal of Surgery. 2021, Elsevier Ltd. p. 13-18
work page 2021
-
[6]
Youssef, S.C., et al., Learning surgical skills through video-based education: a systematic review. Surgical Innovation, 2023. 30(2): p. 220-238
work page 2023
-
[7]
Al Abbas, A.I., et al., The development of a deep learning model for automated segmentation of the robotic pancreaticojejunostomy. Surgical endoscopy, 2024. 38(5): p. 2553-2561
work page 2024
-
[8]
Datta, V ., et al., The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model, in Journal of the American College of Surgeons. 2001. p. 479-485
work page 2001
Show all 45 references
-
[9]
Datta, V ., et al., The relationship between motion analysis and surgical technical assessments, in American Journal of Surgery. 2002. p. 70-73
2002
-
[10]
Surgical Endoscopy, 2021
Ebina, K., et al., Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs. Surgical Endoscopy, 2021. 35(8): p. 4399-4416
2021
-
[11]
Roche, C.A., et al., Kinematic Measures for Evaluating Surgical Skills in Natural Orifice Translumenal Endoscopic Surgery (NOTES), in Studies in Health Technology and Informatics. 2014. p. 339-345
2014
-
[12]
ANZ Journal of Surgery, 2023
Bapna, T., et al., Eye‐tracking in surgery: a systematic review. ANZ Journal of Surgery, 2023. 93(11): p. 2600-2608
2023
-
[13]
Global Surgical Education - Journal of the Association for Surgical Education, 2024
Farah, E., et al., Heart rate variability: an objective measure of mental stress in surgical simulation. Global Surgical Education - Journal of the Association for Surgical Education, 2024. 3(1)
2024
-
[14]
Surgical Endoscopy, 2024
Shafiei, S.B., et al., Classification of subtask types and skill levels in robot-assisted surgery using EEG, eye-tracking, and machine learning. Surgical Endoscopy, 2024. 38(9): p. 5137-5147
2024
-
[15]
2017, Cold Spring Harbor Laboratory
Nemani, A., et al., Assessing bimanual motor skills with optical neuroimaging, in bioRxiv. 2017, Cold Spring Harbor Laboratory. p. 204305
2017
-
[16]
2022, Springer US
Kutana, S., et al., Objective assessment of robotic surgical skills: review of literature and future directions, in Surgical Endoscopy. 2022, Springer US. p. 3698-3707
2022
-
[17]
npj Digital Medicine,
Mascagni, P ., et al., Computer vision in surgery: from potential to clinical value. npj Digital Medicine,
-
[18]
JAMA network open, 2021
Kitaguchi, D., et al., Development and validation of a 3-dimensional convolutional neural network for automatic surgical skill assessment based on spatiotemporal video analysis. JAMA network open, 2021. 4(8): p. e2120786-e2120786
2021
-
[19]
JAMA Netw Open, 2020
Khalid, S., et al., Evaluation of Deep Learning Models for Identifying Surgical Actions and Measuring Performance. JAMA Netw Open, 2020. 3(3): p. e201664
2020
-
[20]
Zia, A. and I. Essa, Automated surgical skill assessment in RMIS training. International journal of computer assisted radiology and surgery, 2018. 13: p. 731-739
2018
-
[21]
arXiv preprint arXiv:1611.05136, 2016
Fard, M.J., et al., Machine learning approach for skill evaluation in robotic-assisted surgery. arXiv preprint arXiv:1611.05136, 2016. 18
2016 arXiv
-
[22]
Langenbeck's Archives of Surgery, 2023
Cheikh Youssef, S., et al., Evolution of the digital operating room: the place of video technology in surgery. Langenbeck's Archives of Surgery, 2023. 408(1)
2023
-
[23]
Sankaranarayanan, and V
Namazi, B., G. Sankaranarayanan, and V . Devarajan, A contextual detector of surgical tools in laparoscopic videos using deep learning. Surgical endoscopy, 2022: p. 1-10
2022
-
[24]
Surgical endoscopy, 2023
Grüter, A.A., et al., Video-based tools for surgical quality assessment of technical skills in laparoscopic procedures: a systematic review. Surgical endoscopy, 2023. 37(6): p. 4279-4297
2023
-
[25]
International Journal of Computer Assisted Radiology and Surgery,
Hutchinson, K., et al., COMPASS: a formal framework and aggregate dataset for generalized surgical procedure modeling. International Journal of Computer Assisted Radiology and Surgery,
-
[26]
Surgical Endoscopy, 2024
Desir, A., et al., Validity and reliability evidence support task-specific metrics for laparoscopic fundoplication. Surgical Endoscopy, 2024. 38(4): p. 2219-2230
2024
-
[27]
Anirudh, and J.V
Diwan, T., G. Anirudh, and J.V . Tembhurne, Object detection using YOLO: Challenges, architectural successors, datasets and applications. multimedia Tools and Applications, 2023. 82(6): p. 9243- 9275
2023
-
[28]
IEEE transactions on medical imaging, 2016
Twinanda, A.P ., et al., Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging, 2016. 36(1): p. 86-97
2016
-
[29]
IEEE Journal of Biomedical and Health Informatics, 2023
Demir, K.C., et al., Deep Learning in Surgical Workflow Analysis: A Review of Phase and Step Recognition. IEEE Journal of Biomedical and Health Informatics, 2023. 27(11): p. 5405-5417
2023
-
[30]
International journal of computer vision, 2008
Russell, B.C., et al., LabelMe: a database and web-based tool for image annotation. International journal of computer vision, 2008. 77: p. 157-173
2008
-
[31]
Segment anything
Kirillov, A., et al. Segment anything. in Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023
2023
-
[32]
Random decision forests
Ho, T.K. Random decision forests. in Proceedings of 3rd international conference on document analysis and recognition. 1995. IEEE
1995
-
[33]
Chen, T. and C. Guestrin. Xgboost: A scalable tree boosting system. in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 2016
2016
-
[34]
Cortes, C. and V . Vapnik, Support-vector networks. Machine learning, 1995. 20: p. 273-297
1995
-
[35]
Koenigstein, and R
Bank, D., N. Koenigstein, and R. Giryes, Autoencoders. Machine learning for data science handbook: data mining and knowledge discovery handbook, 2023: p. 353-374
2023
-
[36]
Applied Sciences, 2022
Fujii, R., et al., Surgical Tool Detection in Open Surgery Videos. Applied Sciences, 2022. 12(20)
2022
-
[37]
Sci Rep, 2021
Lavanchy, J.L., et al., Automation of surgical skill assessment using a three-stage machine learning algorithm. Sci Rep, 2021. 11(1): p. 5197
2021
-
[38]
International journal of computer assisted radiology and surgery, 2019
Ismail Fawaz, H., et al., Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks. International journal of computer assisted radiology and surgery, 2019. 14: p. 1611-1617
2019
-
[39]
Using 3D convolutional neural networks to learn spatiotemporal features for automatic surgical gesture recognition in video
Funke, I., et al. Using 3D convolutional neural networks to learn spatiotemporal features for automatic surgical gesture recognition in video. in International conference on medical image computing and computer-assisted intervention. 2019. Springer
2019
-
[40]
International Journal of Computer Assisted Radiology and Surgery, 2020
Lefor, A.K., et al., Motion analysis of the JHU-ISI gesture and skill assessment working set using robotics video and motion assessment software. International Journal of Computer Assisted Radiology and Surgery, 2020. 15: p. 2017-2025
2020
-
[41]
PLoS one, 2024
Ebina, K., et al., A surgical instrument motion measurement system for skill evaluation in practical laparoscopic surgery training. PLoS one, 2024. 19(6): p. e0305693
2024
-
[42]
Scientific Reports, 2023
Yanik, E., et al., Video-based formative and summative assessment of surgical tasks using deep learning. Scientific Reports, 2023. 13(1)
2023
-
[43]
Advances in Neural Information Processing Systems, 2024
Yang, L., et al., Depth anything v2. Advances in Neural Information Processing Systems, 2024. 37: p. 21875-21911. 19
2024
-
[44]
arXiv preprint arXiv:2410.02073, 2024
Bochkovskii, A., et al., Depth pro: Sharp monocular metric depth in less than a second. arXiv preprint arXiv:2410.02073, 2024
2024 arXiv
-
[2023]
2143-2154
18(12): p. 2143-2154
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.