{"id":"10ad6d89-ded1-4934-b554-9258d970746c","arxiv_id":"2412.16195","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A SAM plus YOLOv8 tool tracker feeding a denoising autoencoder and 1-D CNN classifies novice versus expert laparoscopic suturing from video alone, reaching 0.817 accuracy.","lead":"This paper builds a fully automatic system that watches videos of laparoscopic suturing, tracks the surgical tools, and classifies the surgeon as novice or expert. It reports about 82 percent accuracy using a deep-learning classifier that learns motion patterns directly from the tracked tool paths.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-validation treats 141 suture segments from 49 videos of 28 surgeons as independent; without subject- or video-level grouping the reported accuracies may be inflated by leakage.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the 10-fold cross-validation treats 141 samples as independent although they derive from 49 videos by 28 subjects, with no subject- or video-level grouping. This is the most serious threat to the paper's central numerical claim because the reported accuracy is the primary evidence for 'automated classification independent of human annotation.' The proposed concrete test—subject-grouped cross-validation—is the standard fix and would settle whether the concern lands. I do not see a more fundamental flaw: the tool-tracking validation is indeed circular (the YOLOv8 model is evaluated against masks produced by the same YOLOv8+SAM pipeline), but even if tracking were imprecise, the classification claim would still hinge on the cross-validation independence. The paper is otherwise a plausible integration of known components, so a conditional verdict remains appropriate pending the grouped cross-validation check.","tokens_in":9656,"tokens_out":3393,"duration_ms":32846,"concrete_test":"Re-run the supervised and DAE/1-D CNN classifications using GroupKFold with subject as the group (or, more conservatively, video as the group), ensuring all S1–S3 segments from the same surgeon/video stay in the same fold; stratify by the expert/novice label at the group level. Compare the resulting accuracy and F1 to Tables 1 and 2. If the grouped cross-validation accuracy falls toward chance (or drops substantially below the reported 0.795–0.822), the independence assumption fails and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the pipeline classifies novice vs. expert performance without human annotation, supported by accuracies of 0.795–0.822 (Tables 1 and 2). The load-bearing assumption is that the 141 samples used for the 10-fold cross-validation in Section 2.4.1 are independent. They are not: Section 2.1 reports 49 videos from 28 subjects, and Section 2.3 divides each video into S1–S3 suture segments, yielding up to three samples per video and multiple samples per surgeon. Section 2.4.1 describes only a '10-fold cross-validation stratified split' on the 141 samples, with no stated grouping by subject or by video. If segments from the same video or same surgeon appear in both training and test folds, the classifier can memorize surgeon-specific or video-specific artifacts (e.g., instrument appearance, camera angle, motion style) rather than learn general skill. With only 28 subjects, the effective number of independent units is far smaller than 141, so the reported standard deviations also understate uncertainty. If this leakage is present, the headline accuracy and F1 scores do not establish the claimed annotation-free skill assessment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes a fully automated pipeline for assessing laparoscopic suturing skill from videos of Nissen fundoplication on a porcine model. Tool tracking is obtained by combining a YOLOv8 detector with the Segment Anything Model (SAM) to generate segmentations without manual annotation, followed by kinematic feature extraction and classification into novice versus expert. The authors compare supervised classifiers on engineered kinematic features with an unsupervised denoising autoencoder (DAE) whose learned representations are classified by a 1-D CNN and other models. They report a best supervised accuracy of 0.795 and a best DAE-based accuracy of 0.817, concluding that the system can classify performance independent of human annotation of surgical videos.","tokens_in":9917,"tokens_out":5314,"duration_ms":47329,"significance":"If the results hold, the pipeline would be a useful step toward scalable, low-cost surgical skill assessment, and the integration of SAM with YOLOv8 to generate training labels is a constructive idea that could reduce annotation burden. The comparison between hand-crafted kinematic features and DAE-learned representations is also informative. However, the two central validation claims are currently not established: the tool-tracking accuracy is evaluated against masks generated by the same pipeline rather than human annotations, and the skill-classification accuracy is likely inflated by using suture segments from the same subjects or videos in both training and test folds. The paper is therefore promising but requires substantial re-analysis before the headline conclusion can be accepted.","major_comments":[{"comment":"The 10-fold cross-validation is applied to 141 samples that are not independent. Section 2.1 reports 49 videos from 28 subjects, and Section 2.3 states that each video contributes up to three suture segments (S1, S2, S3) that are used as separate samples. Because the split is described only at the sample level, with no grouping by subject or by video, segments from the same video or surgeon can appear in both training and test folds. This allows the classifier to memorize video- or surgeon-specific artifacts rather than learn general skill, inflating the reported accuracy and F1 scores. The effective number of independent units is 28 subjects, far smaller than 141. Please rerun the evaluation with group-based cross-validation, such as leave-one-subject-out or leave-one-video-out, and report performance with confidence intervals based on the number of subjects.","section":"Section 2.4.1"},{"comment":"The tracking accuracy claim of mAP 0.95 on the fundoplication data is circular. Section 2.2.3 says that 1401 frames were selected because they 'displayed high-quality segmentation with accurately generated masks,' and no human-annotated test set is described. The masks used as ground truth are therefore produced by the same YOLOv8-plus-SAM pipeline that is being validated. This does not support the claim that manual annotation has been eliminated. Please evaluate the tracking model on a held-out set of frames annotated by humans, describe the annotation protocol, and report inter-annotator agreement.","section":"Section 3.1.2"},{"comment":"The abstract states that the unsupervised 1-D CNN achieved superior results with an accuracy of 0.817 and an F1 score of 0.806, but Table 2 shows that Random Forest achieved a higher accuracy (0.822), higher positive predictive value (0.845), and higher negative predictive value (0.887) when used with DAE features. The 1-D CNN is better only on F1 score. Please specify the primary outcome used for model selection and qualify the 'superior' claim accordingly, or the abstract should be revised to reflect the metric-dependent nature of the comparison.","section":"Abstract and Table 2"},{"comment":"The manuscript reports 141 samples for the supervised machine-learning classification but 196 samples for the DAE pipeline. The relationship between these two sample sets is not explained, making it impossible to determine whether the two approaches are evaluated on the same cohort and whether the comparison in Tables 1 and 2 is fair. Please clarify how the 196 DAE samples are constructed (e.g., concatenation of S1-S3 for each tool and axis) and reconcile the counts with the 141-sample dataset.","section":"Sections 2.4.1 and 2.4.2"},{"comment":"The exclusion of 11 videos due to incomplete tasks or camera instability, and the exclusion of S4 segments because they were performed by only 2-3 subjects, is reported without checking whether these exclusions are balanced across the novice and expert groups. If excluded videos are disproportionately from one skill group, the reported accuracies may reflect a biased sample. Please report exclusions by group and, if feasible, run a sensitivity analysis including the excluded videos or segments.","section":"Sections 2.1 and 2.3"}],"minor_comments":[{"comment":"The term 'unsupervised' is applied to the 1-D CNN, but the final classification uses PGY-level labels; only the DAE feature-learning stage is unsupervised. Please revise the wording to avoid implying that skill labels are not used.","section":"Abstract"},{"comment":"The denoising autoencoder is cited as [25,26], but reference [25] is the COMPASS paper and reference [26] is Desir et al.; neither appears to describe DAE. Please check and correct the citations.","section":"Section 2.4.2"},{"comment":"The plus/minus values are not defined. Please state whether they are standard deviations across the 10 folds and report 95% confidence intervals given the small number of subjects.","section":"Tables 1 and 2"},{"comment":"The sentence 'Only 1401 frames that displayed high-quality segmentation with accurately generated masks were selected' introduces a selection step that may bias the training data. Please describe how 'high-quality' was determined and whether the selection was automated or involved human review.","section":"Section 2.2.3"},{"comment":"The caption says 'Raw Data versus Filtered Data,' but the axes are not labeled. Please label the x- and y-axes and state what the plotted traces represent.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The central weakness is the likely data leakage in the cross-validation, which directly affects the headline accuracies, and the circular validation of the tracking model. Both are fixable in principle with grouped cross-validation and a small human-annotated tracking test set. If the authors cannot provide such re-analyses, the paper's main claims would not be supported. The sample-count inconsistency between Sections 2.4.1 and 2.4.2 also needs clarification before the comparison between approaches can be assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague, quick take on 2412.16195. The paper is a plausible engineering contribution, not a breakthrough. What is genuinely new: an integrated pipeline that uses YOLOv8 (fine-tuned on Cholec80) to propose boxes, SAM to produce masks, retrains a detector on a new fundoplication dataset, and then classifies novices vs experts either from kinematic features or from a DAE plus 1-D CNN. The fundoplication dataset itself (49 videos, 28 subjects) is a useful new resource. The paper is clearly written, the supervised-versus-unsupervised ablation is helpful, and the authors openly acknowledge the 2D limitation and single-task scope. The comparison with prior work is fair.\n\nThe problems are load-bearing. The 10-fold CV in Section 2.4.1 is described only as a stratified split on the 141 samples. Those samples come from S1-S3 segments of 49 videos by 28 subjects. Without subject- or video-level grouping, segments from the same surgeon can appear in both training and test folds, and the classifier can memorize surgeon-specific artifacts. With 28 subjects, the effective sample size is far below 141, so the reported 0.795-0.822 accuracies and the small standard deviations are likely optimistic. The stress-test note lands here.\n\nSecond, the tracking validation is circular: Section 3.1.2 reports mAP 0.95 on fundoplication data using masks generated by the same YOLOv8+SAM pipeline (Section 2.2.3). There is no human-annotated test set. The abstract's claim of being \"independent of human annotation\" is also too strong: Cholec80 labels are human annotations, Section 2.2.1 mentions LabelMe conversions, and the 1401 selected frames were chosen for \"high-quality segmentation,\" which is a manual quality filter.\n\nMinor issues: 141 samples are used for the ML models and 196 for the DAE without reconciling the numbers; no code or data is released; and model comparisons lack a statistical test. None of these are fatal by themselves.\n\nThe paper deserves a serious referee, not a desk reject, because the component integration is new and the dataset is useful. The revision should be conditional: subject-level cross-validation and a human-labeled tracking test set are needed. If those do not change the numbers, the main claim falls; if they survive, this is a modest but solid contribution. I would not cite it for the accuracy figures until that is done.","headline":"Useful integrated pipeline and new dataset, but the headline accuracy and 'no human annotation' claim are not yet supported: likely CV leakage and circular tracking validation.","tokens_in":10468,"tokens_out":2699,"would_cite":false,"duration_ms":24674,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An automated video-analysis pipeline tracks both laparoscopic instruments, extracts their motion, and classifies the surgeon as novice or expert with up to 0.817 accuracy and an F1 of 0.806, using no human-annotated surgical videos.","keywords":["automated surgical skill assessment","laparoscopic fundoplication","tool tracking","Segment Anything Model","YOLOv8","kinematic features","denoising autoencoder","novice/expert classification"],"falsifier":"Re-run the skill classification with a leave-one-video-out or leave-one-surgeon-out cross-validation; if accuracy falls substantially below the reported 0.817, the human-annotation-free claim only holds for seen videos.","tokens_in":9430,"feed_emoji":"🩺","tokens_out":6799,"duration_ms":58703,"temperature":0.7,"pith_summary":"This paper tries to establish that surgical skill assessment can be fully automated: a video-only pipeline locates and tracks both instruments, extracts their motion, and classifies the surgeon as novice or expert without any human annotation. It also compares two modeling strategies: supervised classifiers on kinematic features and an unsupervised denoising autoencoder that learns features directly from tool trajectories. The unsupervised 1-D CNN performs best, with accuracy 0.817 ± 0.108 and F1 0.806 ± 0.110. If the result holds, trainees could receive immediate objective feedback from ordinary surgical recordings instead of waiting for expert video review.","feed_headline":"No-label AI scores suturing skill at 82 percent accuracy","feed_subtitle":"Automatic tool tracking plus an unsupervised network separates expert from novice surgeons with no manual video labeling.","key_machinery":"The load-bearing mechanism is the automated annotation and tracking chain: YOLOv8 produces approximate bounding boxes for surgical tools, the Segment Anything Model converts those boxes into precise segmentation masks, and a retrained detection model tracks the tooltips through the video; a low-pass Butterworth filter at 24 Hz smooths the trajectories. The classification then proceeds through two streams: a supervised stream that computes RMS velocity, RMS acceleration, RMS jerk, total path length, and bimanual dexterity, and an unsupervised stream that renders x-y tooltip trajectories as line-plot images, trains a denoising autoencoder, and classifies the resulting 128-dimensional encodings with a 1-D CNN.","core_discovery":"The central claim is that a fully automated video-analysis pipeline can grade laparoscopic suturing skill at a level competitive with human annotation, using no manually labeled surgical videos. The pipeline combines a detector that proposes tool bounding boxes with a segmentation model that refines them into precise instrument masks; it then tracks tooltips, filters the trajectories at 24 Hz, and feeds the resulting motion into either handcrafted kinematic features or learned representations. On 141 suturing segments from 28 surgeons performing Nissen fundoplication on porcine bowel, the best result is the unsupervised 1-D CNN with accuracy 0.817 ± 0.108 and F1 0.806 ± 0.110, which the paper presents as superior to supervised feature-based classification with 0.795 accuracy.","pith_inferences":["A direct consequence the paper does not test: segment-level cross-validation may overstate generalizability; a leave-one-video-out split would reveal whether accuracy survives seeing a surgeon's entire video only at test time.","Extending the pipeline with monocular depth estimation, which the paper flags as future work, could separate true instrument motion from camera motion and improve the skill signal.","The DAE's learned 128-dimensional encodings might recapitulate the handcrafted kinematic features; comparing them directly would clarify what the unsupervised model actually learns."],"forward_implications":["Trainees could get immediate, automatic skill labels after each suturing segment, removing the bottleneck of expert video review.","The unsupervised DAE route shows that hand-crafted kinematic metrics are not necessary, simplifying deployment to new tasks.","Because the motion is extracted from ordinary 2-D laparoscopic video, the approach avoids external sensors and robotic-system kinematics.","With additional fine-tuning of the tool detector, the pipeline could transfer to other laparoscopic procedures where instrument types differ."],"supporting_citations":[{"why":"Supplies the Segment Anything Model used to convert YOLOv8 bounding boxes into precise tool masks.","marker":"[31]"},{"why":"Supplies the Cholec80 dataset and pretraining basis for surgical tool detection with YOLOv8.","marker":"[28, 29]"},{"why":"Supplies the fundoplication dataset and the novice/expert performance groups used in this study.","marker":"[26]"},{"why":"Serves as the prior motion-analysis baseline on annotated surgical data that this work extends from descriptive statistics to classification.","marker":"[40]"},{"why":"Provides the fully convolutional network on kinematic data that motivates the learned-features approach.","marker":"[38]"},{"why":"Provides the 3-D convolutional video model that requires pre-segmented clips, the comparison point for the fully automated pipeline.","marker":"[39]"},{"why":"Shows a denoising autoencoder assessing an FLS pattern-cutting task, the direct precedent for the DAE classifier.","marker":"[42]"},{"why":"Supplies the general autoencoder formulation that the DAE design builds on.","marker":"[35]"}],"fun_headline_variants":["Unsupervised AI grades suturing skill at 82% accuracy","No-label AI matches expert suturing assessment at 82%","AI auto-scores laparoscopic suturing without manual labels","82% accuracy: AI evaluates suturing from video alone","Self-taught AI scores suturing skill, zero human labeling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the 141 suturing segments can be treated as independent samples even though many come from the same video and the same surgeon; if folds mix segments from one video, the reported accuracy could reflect memorization rather than general skill.","fun_headline_variants_meta":{"raw":{"variants":["Unsupervised AI grades suturing skill at 82% accuracy","No-label AI matches expert suturing assessment at 82%","AI auto-scores laparoscopic suturing without manual labels","82% accuracy: AI evaluates suturing from video alone","Self-taught AI scores suturing skill, zero human labeling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000764,"raw_usage":{"total_tokens":3443,"prompt_tokens":1054,"completion_tokens":2389,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":2305}},"tokens_in":670,"tokens_out":2389,"duration_ms":16379,"temperature":1.0,"reasoning_tokens":2305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:21:28.604507+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the skill classification with a leave-one-video-out or leave-one-surgeon-out cross-validation; if accuracy falls substantially below the reported 0.817, the human-annotation-free claim only holds for seen videos.","supporting_citations":[{"cited_title":"Segment anything","cited_arxiv_id":null,"evidence_quote":"Supplies the Segment Anything Model used to convert YOLOv8 bounding boxes into precise tool masks."},{"cited_title":"Surgical Endoscopy, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the fundoplication dataset and the novice/expert performance groups used in this study."},{"cited_title":"International Journal of Computer Assisted Radiology and Surgery, 2020","cited_arxiv_id":null,"evidence_quote":"Serves as the prior motion-analysis baseline on annotated surgical data that this work extends from descriptive statistics to classification."},{"cited_title":"International journal of computer assisted radiology and surgery, 2019","cited_arxiv_id":null,"evidence_quote":"Provides the fully convolutional network on kinematic data that motivates the learned-features approach."},{"cited_title":"Using 3D convolutional neural networks to learn spatiotemporal features for automatic surgical gesture recognition in video","cited_arxiv_id":null,"evidence_quote":"Provides the 3-D convolutional video model that requires pre-segmented clips, the comparison point for the fully automated pipeline."},{"cited_title":"Scientific Reports, 2023","cited_arxiv_id":null,"evidence_quote":"Shows a denoising autoencoder assessing an FLS pattern-cutting task, the direct precedent for the DAE classifier."},{"cited_title":"Koenigstein, and R","cited_arxiv_id":null,"evidence_quote":"Supplies the general autoencoder formulation that the DAE design builds on."}],"review_version":1}