General surgery vision transformer: A video pre-trained foundation model for general surgery
read the original abstract
The absence of openly accessible data and specialized foundation models is a major barrier for computational research in surgery. Toward this, (i) we open-source the largest dataset of general surgery videos to-date, consisting of 680 hours of surgical videos, including data from robotic and laparoscopic techniques across 28 procedures; (ii) we propose a technique for video pre-training a general surgery vision transformer (GSViT) on surgical videos based on forward video prediction that can run in real-time for surgical applications, toward which we open-source the code and weights of GSViT; (iii) we also release code and weights for procedure-specific fine-tuned versions of GSViT across 10 procedures; (iv) we demonstrate the performance of GSViT on the Cholec80 phase annotation task, displaying improved performance over state-of-the-art single frame predictors.
This paper has not been read by Pith yet.
Forward citations
Cited by 6 Pith papers
-
SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery
SurgAtlas is a new dataset of 15,291 surgical videos totaling 2,391 hours with multi-level annotations that supports finetuning models to competitive performance on surgical benchmarks.
-
SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark
SurgCoT is a new benchmark that evaluates chain-of-thought spatiotemporal reasoning in multimodal large language models on surgical videos using five defined dimensions and an annotation protocol of Question-Option-Kn...
-
Federated Learning for Surgical Vision in Appendicitis Classification: Results of the FedSurg EndoVis 2024 Challenge
The FedSurg challenge benchmarks federated learning on appendectomy videos and finds only 26% F1 on unseen centers even with centralized data, plus extra penalties from decentralization, with spatiotemporal models per...
-
SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos
SurgMotion outperforms prior methods on 17 surgical video benchmarks by shifting pretraining to latent motion prediction with motion-guided masking, affinity distillation, and diversity regularization on a 15M-sample dataset.
-
HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Space
HyperVLP uses hyperbolic geometry in surgical video-language pre-training to preserve hierarchy across actions, steps, and phases, yielding gains in zero- and few-shot phase recognition.
-
Surgical Anatomy Recognition with Context Learning using Foundation Representations
Presents ATLAS-120k dataset and ATLAS model for context-aware surgical anatomy segmentation using foundation representations and temporal cues.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.