Pith. sign in

REVIEW 4 major objections 7 minor 3 cited by

SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SurgVLM claims that a single vision-language model fine-tuned on a unified multimodal surgical database can handle ten surgical tasks and outperform fourteen commercial vision-language models on every metric of a new six-dataset benchmark.

desk verdict Valuable dataset and benchmark, but the headline superiority over commercial VLMs is in-distribution fine-tuning, not general surgical intelligence. read the letter →

arxiv 2506.02555 v1 pith:B2RJGFV3 submitted 2025-06-03 cs.CV

Zhitao Zeng , Zhu Zhuo , Xiaojun Jia , Erli Zhang , Junde Wu , Jiaan Zhang , Yuxuan Wang , Chang Han Low
show 7 more authors
Jian Jiang Zilong Zheng Xiaochun Cao Yutong Ban Qi Dou Yang Liu Yueming Jin
This is my paper
classification cs.CV
keywords surgicalvision-languagemodelintelligencefoundationmultimodaldatabasephaserecognitionactiontripletcriticalviewofsafetyinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SurgVLM is a family of vision-language models built on Qwen2.5-VL and fine-tuned on a newly assembled multimodal surgical database, SurgVLM-DB, which folds 23 public datasets (16 surgical types, 18 anatomical structures, over 1.8 million frames, 7.79 million conversations) into ten surgical tasks organized in a perception-to-reasoning hierarchy. The paper's claim is that one universal model can handle all ten tasks — instrument and tissue recognition and localization, phase, step, action and triplet recognition, and critical-view-of-safety assessment — and that it outperforms 14 mainstream commercial vision-language models on every metric of a new six-dataset benchmark, SurgVLM-Bench. The headline result is an overall arena score of 336.21 for SurgVLM-72B against 191.70 for Gemini 2.0 Flash, a 75.4% gain, with especially large jumps on triplet recognition and instrument localization. The authors argue the gains come primarily from the database's four-stage construction pipeline — label cleaning, cross-task correlation enrichment, explainable answer generation, and conversational diversity expansion — rather than from new architecture, and that a mid-sized SurgVLM-32B already captures most of the benefit on recognition tasks.

What carries the argument

The load-bearing mechanism is the SurgVLM-DB construction pipeline feeding a hierarchical vision-language alignment objective. The four-stage pipeline (1) cleans and standardizes labels to professional terminology, (2) merges correlated attributes such as phase–step and instrument–action into joint prompts, (3) augments answers with explainable, vision-grounded descriptions, and (4) expands 100–200 conversation formats per task with single- and multi-turn variants. The ten tasks are organized into three tiers — visual perception (five tasks), temporal analysis (four tasks), and reasoning (critical-view-of-safety assessment) — and trained with a single unified autoregressive loss so the model learns task dependencies instead of isolated labels. The model itself follows the Qwen2.5-VL architecture with dynamic-resolution patch encoding, windowed attention, M-RoPE positional encoding aligned to real timestamps, and 4x visual token compression.

What would settle it

Retrain SurgVLM-72B on SurgVLM-DB with the training splits of all six SurgVLM-Bench datasets removed, then evaluate on the same benchmark; if the arena score falls toward the 118–192 range of the commercial baselines, the reported 75% margin reflects in-distribution adaptation rather than transferable surgical understanding. A complementary check is zero-shot evaluation on a procedure type absent from the 16 in SurgVLM-DB, comparing SurgVLM-72B against Gemini 2.0 Flash under the same open-vocabulary protocol.

Watch

Extended reading notes

Core claim

The paper sets out to establish that a single vision-language foundation model can carry the full range of surgical-intelligence tasks once it is given enough structured, hierarchically organized domain supervision. SurgVLM-DB supplies that supervision: labels from 23 public datasets are cleaned to professional medical terminology, paired across correlated tasks (phase with step, instrument with action), expanded into explainable descriptions that tie text to visual evidence, and rendered into 100–200 varied single- and multi-turn conversation templates per task. The models are then trained with a single unified autoregressive loss over all ten tasks, with size scaled from 7B to 32B to 72B under progressively lighter fine-tuning (full fine-tuning, frozen backbone with adapters, LoRA). On SurgVLM-Bench the authors report consistent superiority across all 24 metrics: phase recognition accuracy rises from 38.89% (Gemini 2.0 Flash) to 76.40% (SurgVLM-72B open-vocabulary), triplet accuracy from 2.08% to 13.10%, instrument localization mIoU from 19.90% to 59.40%, and critical-view-of-safety assessment from 59.61% to 76.73%. The authors also report that scaling behaves differently by task: localization keeps improving with model size, recognition plateaus near 30B parameters, and CVS stays flat, which they read as evidence that the bottleneck for surgical adaptation is data quality and task structure rather than backbone capacity.

Load-bearing premise

The comparison assumes that beating untrained commercial models on benchmark datasets whose training splits are included in SurgVLM-DB demonstrates superior surgical intelligence, rather than successful memorization of those datasets' label distributions.

Editorial extensions

If this is right

  • A single fine-tuned SurgVLM can replace multiple task-specific surgical AI pipelines: the same weights handle perception, temporal analysis, and safety reasoning, with open-vocabulary responses rather than fixed-choice answers.
  • On the authors' benchmark, commercial VLMs are far behind — GPT-4o scores 118.71 against SurgVLM-72B's 336.21 — implying general-purpose multimodal models lack the surgical vocabulary and spatial grounding needed for clinical use.
  • The reported scaling pattern implies that a mid-sized model around 30B parameters with adapter tuning reaches competitive recognition and CVS performance at a fraction of the cost of a 72B model, so surgical deployment does not require the largest model.
  • The data pipeline is transferable: the same cleaning-correlation-explanation-diversification recipe could be applied to other procedurally annotated medical video corpora to build domain-specific VLMs without new architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark is in-distribution: all six SurgVLM-Bench datasets appear in SurgVLM-DB, so the comparison against commercial VLMs measures fine-tuned adaptation on seen task distributions rather than zero-shot transfer; a genuinely out-of-distribution evaluation on a procedure type absent from the 16 included would likely shrink the reported margin.
  • The exact-keyword open-vocabulary protocol penalizes synonyms and paraphrases, which may understate commercial VLMs; a paraphrase-tolerant scorer would give a fairer picture of general models while arguably keeping the precision bar that clinical use requires.
  • A direct ablation of the four pipeline modules — for instance, removing cross-task correlation enrichment while keeping everything else identical — would attribute the reported gains and is the natural next experiment the current study leaves implicit.
  • If the gains hold under out-of-distribution testing, the practical consequence is that hospitals could adapt existing open VLMs to their own surgical video archives with modest compute, since the authors find no need for a surgical-specific vision backbone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This manuscript proposes SurgVLM, a family of surgical vision-language models (7B/32B/72B) built on Qwen2.5-VL, together with a large multimodal surgical database SurgVLM-DB (approximately 1.81M frames and 7.79M conversations assembled from 23 public datasets) and a six-dataset benchmark SurgVLM-Bench. The authors report that SurgVLM-72B achieves an arena score of 336.21 versus 191.70 for Gemini 2.0 Flash, and claim consistent improvements over 14 commercial VLMs across all 24 metrics. The central comparison is confounded: all six benchmark datasets are included in SurgVLM-DB, so the reported superiority reflects in-distribution supervised fine-tuning rather than a head-to-head test of general surgical intelligence against zero-shot commercial models. The evaluation protocol also contains internal inconsistencies in benchmark sampling and in the metrics reported across protocols.

Significance. The dataset contribution is potentially substantial: SurgVLM-DB unifies 23 public datasets across 16 surgical types and provides a task hierarchy spanning perception, temporal analysis, and reasoning, and SurgVLM-Bench covers six widely used tasks. If the evaluation were made fair, this would be a valuable community resource. However, the headline performance claims are not currently supported because the benchmark is drawn from the training database and the baselines are evaluated zero-shot; the paper provides no same-base fine-tuned control and no holdout evaluation on surgical data absent from SurgVLM-DB. The claimed 75.4% improvement over Gemini 2.0 Flash is therefore, at best, evidence of successful domain adaptation to the benchmark distribution rather than of superior surgical intelligence.

major comments (4)
  1. [Section 2.3, Supplement C, Table S2] The central comparison is confounded by train/evaluation overlap. Table S2 explicitly lists EndoVis2017, EndoVis2018-VQA, Cholec80, CholecT50, SAR-RARP, and Endoscapes2023 as components of SurgVLM-DB, with nonzero test counts, and Supplement C states that SurgVLM-Bench is selected from SurgVLM-DB and that frames are sampled from the official test split of each dataset. Since all SurgVLM variants are SFT on SurgVLM-DB, the arena-score gap in Table 2 may be inflated by direct test-frame contamination or, if only train splits were used, by in-distribution adaptation. The commercial baselines are evaluated zero-shot, so the comparison does not support the claim of superior surgical intelligence. Provide a control where the same base model (e.g., Qwen2.5-VL-72B-Instruct) is LoRA-tuned on SurgVLM-DB and evaluated under the identical protocol, and/or evaluate on a surgical dataset not present in SurgVLM-DB; otherwise the superiority claims should be reframed as domain-adaptation results.
  2. [Supplement C] The stated benchmark sampling protocol is internally inconsistent. Supplement C says that 1000 frames are randomly sampled from the official test split of each dataset, but Table S2 reports EndoVis2017 test count as 600 and EndoVis2018-VQA test count as 447. Clarify the exact number of frames actually used per dataset and report per-dataset sample sizes in the main text, since the arena score sums normalized scores and unequal sampling affects the comparability of the aggregate metric.
  3. [Section 2.3, Tables 3-4] The claim that SurgVLM 'consistently outperforms commercial VLMs across all 24 metrics' mixes evaluation protocols. In Tables 3 and 4, all commercial baselines are evaluated under MCQ, while SurgVLM-7B and SurgVLM-32B are reported only under OV, with SurgVLM-72B reported under both protocols. The paper should state unambiguously which model/protocol combination underlies each headline claim and present same-protocol results for every model; the exact-match keyword rule for OV and the MCQ option-selection rule are not interchangeable, so mixing them makes the aggregate comparison difficult to verify.
  4. [Section 2.1, Table 1, Abstract] The reported scale of SurgVLM-DB is inconsistent: the abstract and introduction say 'over 1.81 million frames,' Table 1 says '1810K' images, and Section 2.1 says '1.181M annotated images.' Correct the mismatch, since dataset scale is a primary claimed contribution.
minor comments (7)
  1. [Section 2.3] The sentence 'SurgVLM-72B under MCQ evaluation delivers an additional 4.35 point boost over its MCQ counterpart by demonstrating true OV reasoning' appears to be a typo: the comparison is between OV and MCQ evaluation of the same 72B model, not between two MCQ runs.
  2. [Supplement C.1] The text refers to 'SurgX-MMBench' in the evaluation metrics section; this appears to be a leftover from a different benchmark template and should be replaced with SurgVLM-Bench.
  3. [Table 3] The spelling 'Endovis2018' is inconsistent; elsewhere the paper uses 'EndoVis2018.' Please standardize dataset names throughout.
  4. [Table 3] The model name 'MiniCPM-O-2 6' is missing a dot; it should be 'MiniCPM-O-2.6'.
  5. [Section 2.4] The text states that Cholec80 phase recognition accuracy 'notably decreases to 69.99%' with SurgVLM-72B MCQ, but Table 3 reports 69.66%; the numbers should be reconciled.
  6. [Figure 3a] The labels 'Improve 15.62%' and 'Improve 51.68%' do not state the reference model for the improvement; specify what each percentage is relative to.
  7. [Section 2.2] The sentence 'All datasets in SurgVLM-Bench employ the official training/testing splits and are balanced in terms of surgical types and anatomical structures, ensuring a comprehensive, end-to-end evaluation of multimodal surgical intelligence without fine-tuning' should clarify that only the baselines are evaluated without fine-tuning, since the SurgVLM models are fine-tuned on SurgVLM-DB.

Circularity Check

1 steps flagged · score 6.0 of 10

SurgVLM-Bench is a subset of the SurgVLM-DB training database, so SurgVLM's reported superiority over zero-shot commercial VLMs reflects in-distribution fine-tuning rather than an independent test of surgical intelligence.

  1. fitted input called prediction [Supplement C (Benchmark Construction); Section 2.3 (Performance Comparison); Table S2]
    "From SurgVLM-DB, we selected six datasets to build SurgVLM-Bench ... we randomly sample 1000 frames from the official test split in each dataset ... All SurgVLM models are Supervised Fine-tuning (SFT) exclusively on SurgVLM-DB, achieving substantially higher arena scores..."

    The six SurgVLM-Bench datasets are all listed in Table S2 as part of SurgVLM-DB, and Supplement C samples the benchmark frames from the official test splits of those datasets. SurgVLM is then fine-tuned on SurgVLM-DB, so the benchmark measures performance on the same datasets and label distributions used for training. The comparison is made against zero-shot commercial VLMs, with no same-data fine-tuned baseline, so the reported 75.4% arena-score gain over Gemini 2.0 Flash is not evidence of general surgical intelligence; it is at least partly the expected result of evaluating on the training distribution.

full rationale

The paper's central performance claim is the arena-score comparison on SurgVLM-Bench. All six SurgVLM-Bench datasets are explicitly part of SurgVLM-DB (Table S2), and Supplement C says SurgVLM-Bench randomly samples 1000 frames from each dataset's official test split. Since SurgVLM is SFT exclusively on SurgVLM-DB, the evaluation is in-distribution: the model has been adapted to the same datasets and task schemas that define the benchmark. The zero-shot commercial baselines do not control for fine-tuning on the benchmark distribution, so the headline 'consistently surpasses 14 mainstream commercial VLMs across all 24 metrics' is not an independent test of surgical intelligence. Table S2's use of 'dev/test' counts inside SurgVLM-DB raises the additional possibility that test frames themselves were included in training, but the core circularity does not depend on that ambiguity: even with strictly held-out splits, the benchmark is a subset of the training database, making the evaluation a measure of domain adaptation rather than generalization. There are no load-bearing self-citations or imported uniqueness theorems; the database construction and model design are independent contributions. The circularity is confined to the evaluation design, which partially forces the reported superiority, so a moderate score of 6 is appropriate.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper's claims rest on the validity of the assembled public datasets, the correctness of GPT-generated explanations, and the fairness of a benchmark that overlaps with the training distribution. These assumptions are plausible but not independently verified, and the evaluation design limits the strength of the central claim.

free parameters (3)
  • Exact-match keyword evaluation rule = exact ground-truth keyword required
    In the OV protocol, responses must contain the exact ground truth keyword and paraphrases are marked incorrect (Section 2.2). This hand-chosen rule disadvantages general VLMs and shapes the reported accuracy.
  • Evaluation frame sampling = 1000 frames per dataset (stated)
    Supplement Section C: 'we randomly sample 1000 frames from the official test split in each dataset'. This is inconsistent with reported test set sizes, e.g., EndoVis2017 has 600 test frames, and affects measured scores.
  • Arena score normalization = per-dataset normalized to 0-100, then summed
    The arena score (Section 2.2) is a hand-chosen composite; the normalization weights and per-dataset scaling are not specified, making cross-model comparisons hard to interpret.
assumptions (3)
  • domain assumption Public surgical datasets provide reliable ground-truth labels for training and evaluation
    SurgVLM-DB is built from 23 public datasets (Table S2) and SurgVLM-Bench uses six of them; if labels are noisy or inconsistent, reported performance is affected. Invoked throughout Section 2.1.
  • domain assumption GPT-generated explainable answers are semantically correct expansions of the original labels
    Module 3 (Explainable Answer Generation) uses GPT to enrich annotations; errors in the generated explanations would propagate into the training data. Described in Section 2.1, Module 3.
  • ad hoc to paper The task hierarchy (perception, temporal analysis, reasoning) is the correct ordering for curriculum-style alignment
    The hierarchy is asserted as domain knowledge (Figure 2b) and used to structure training data, but no evidence is given that this ordering improves learning over alternative arrangements.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence." pith.science (2026). https://pith.science/paper/B2RJGFV3

@misc{pith2026250602555,
  author       = {Pith},
  title        = {Pith review of: SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B2RJGFV3}},
  note         = {Machine review of arXiv:2506.02555}
}
read the original abstract

Foundation models have achieved transformative success across biomedical domains by enabling holistic understanding of multimodal data. However, their application in surgery remains underexplored. Surgical intelligence presents unique challenges - requiring surgical visual perception, temporal analysis, and reasoning. Existing general-purpose vision-language models fail to address these needs due to insufficient domain-specific supervision and the lack of a large-scale high-quality surgical database. To bridge this gap, we propose SurgVLM, one of the first large vision-language foundation models for surgical intelligence, where this single universal model can tackle versatile surgical tasks. To enable this, we construct a large-scale multimodal surgical database, SurgVLM-DB, comprising over 1.81 million frames with 7.79 million conversations, spanning more than 16 surgical types and 18 anatomical structures. We unify and reorganize 23 public datasets across 10 surgical tasks, followed by standardizing labels and doing hierarchical vision-language alignment to facilitate comprehensive coverage of gradually finer-grained surgical tasks, from visual perception, temporal analysis, to high-level reasoning. Building upon this comprehensive dataset, we propose SurgVLM, which is built upon Qwen2.5-VL, and undergoes instruction tuning to 10+ surgical tasks. We further construct a surgical multimodal benchmark, SurgVLM-Bench, for method evaluation. SurgVLM-Bench consists of 6 popular and widely-used datasets in surgical domain, covering several crucial downstream tasks. Based on SurgVLM-Bench, we evaluate the performance of our SurgVLM (3 SurgVLM variants: SurgVLM-7B, SurgVLM-32B, and SurgVLM-72B), and conduct comprehensive comparisons with 14 mainstream commercial VLMs (e.g., GPT-4o, Gemini 2.0 Flash, Qwen2.5-Max).

Figures

Figures reproduced from arXiv: 2506.02555 by the authors.

Figure 1
Figure 1. Illustration of our Surgical Multimodal database SurgVLM-DB. a, SurgVLM-DB contains 16 surgical types and 18 anatomical structures, reflecting the broad diversity of SurgVLM-DB. b, SurgVLM-DB contain 1.181M annotated images with 7.79M conversations, and the distribution of video numbers demonstrates large-scale comprehensive data in various surgical types, ensuring the robustness of SurgVLM [PITH_FULL_IMAGE:figures… view at source ↗
Figure 2
Figure 2. Illustration of our Surgical Multimodal Database, SurgVLM-DB. a, Construction pipeline can be divided into four modules. b, Overview of task hierarchy in SurgVLM-DB containing 10 surgical tasks, ranging from visual perception to temporal analysis to reasoning. c, Comparison showing SurgVLM-DB has the largest number of images, conversations, and surgical types indicated by the size of bubble to our knowledge. d, Dist… view at source ↗
Figure 3
Figure 3. Comparison of SurgVLM and 14 mainstream commercial VLMs. a, Leaderboard on SurgVLM-Bench by overall arena score, demonstrating superior performance of our SurgVLM compared with other mainstream commercial VLMs. b, Comprehensive comparison with Gemini 2.0 Flash, Qwen2.5 Max, and GPT-4o. SurgVLM consistently outperforms commercial VLMs across 24 metrics. c, Detailed comparison with 14 mainstream commercial VLMs on the… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison with typical examples, of SurgVLM-72B, Gemini 2.0 Flash, Qwen 2.5 Max, and GPT-4o, demonstrating the superior performance of SurgVLM. Green texts indicate correct response, while red texts represent wrong response. Security checking with purple c…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A retrieval-based surgical video model that searches a surgery-specific concept vocabulary achieves state-of-the-art zero-shot results on most benchmarks at a fraction of generative latency.

  2. MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    On MarineEVT, an event-centric 20K-pair marine video QA benchmark, EVT-R1 with tool-integrated RL scores 48.89 average accuracy, 5.22 points above the best untuned open-source VLM and 8.54 above the best tool-using co...

  3. LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Latent-action modeling (inverse dynamics plus forward world model) with a patch-level anti-collapse regularizer improves surgical action-triplet recognition and makes encoder change features land more on instrument-ti...

Reference graph

Works this paper leans on

88 extracted references · 50 canonical work pages · cited by 3 Pith papers

  1. [1]

    Phi-4-mini technical report: Com- pact yet powerful multimodal language models via mixture-of-loras

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Con- gcong Chen, et al. Phi-4-mini technical report: Com- pact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025. 4, 9, 11, 12

    arXiv 2025

  2. [2]

    Cholecinstanceseg: A tool instance segmen- tation dataset for laparoscopic surgery, 2024

    Oluwatosin Alabi, Ko Ko Zayar Toe, Zijian Zhou, Char- lie Budd, Nicholas Raison, Miaojing Shi, and Tom Ver- cauteren. Cholecinstanceseg: A tool instance segmen- tation dataset for laparoscopic surgery, 2024. 4, 23, 24, 25

    2024

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Ro- man Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Ja- cob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj ...

    2022

  4. [4]

    2017 robotic instrument segmentation and track- ing challenge

    Max Allan, Alexey Shvets, Thomas Kurmann, Rui Zhang, Rajesh Duggal, Yuhang Su, Sebastian Bodenst- edt, Stefanie Speidel, Sebastian Kletz, Pietro Mascagni, et al. 2017 robotic instrument segmentation and track- ing challenge. In arXiv preprint arXiv:1902.06426, 2019. 4, 6, 7, 23, 24, 26

    arXiv 2017

  5. [5]

    Pixel-wise recognition for holistic surgical scene under- standing

    Nicol ´as Ayobi, Santiago Rodr ´ıguez, Alejandra P ´erez, Isabela Hern ´andez, Nicol ´as Aparicio, Eug ´enie Des- sevres, Sebasti ´an Pe ˜na, Jessica Santander, Juan Igna- cio Caicedo, Nicol ´as Fern ´andez, and Pablo Arbel ´aez. Pixel-wise recognition for holistic surgical scene under- standing. In Proceedings of the IEEE/CVF International Conference on ...

  6. [6]

    Surgical-vqla: Transformer with gated vision-language embedding for visual ques- tion localized-answering in robotic surgery

    Long Bai, Mobarakol Islam, Lalithkumar Seenivasan, and Hongliang Ren. Surgical-vqla: Transformer with gated vision-language embedding for visual ques- tion localized-answering in robotic surgery. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 6859–6865. IEEE, 2023. 2

    2023

  7. [7]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wen- bin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 4, 7, 9, 11, 12, 13

    arXiv 2025

  8. [8]

    Curriculum learning

    Yoshua Bengio, J ´erˆome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning , pages 41–48. ACM, 2009. 15

    2009

Show all 88 references
  1. [9]

    De- tecting surgical tools by modelling local appearance and global shape

    David Bouget, Rodrigo Benenson, Mohamed Omran, Laurent Riffaud, Bernt Schiele, and Pierre Jannin. De- tecting surgical tools by modelling local appearance and global shape. IEEE transactions on medical imaging, 34(12):2603–2617, 2015. 4

    2015

  2. [10]

    Rinner, Sebastian Bo- denstedt, Alexander C

    Matthias Carstens, Franziska M. Rinner, Sebastian Bo- denstedt, Alexander C. Jenke, J ¨urgen Weitz, Marius Distler, Stefanie Speidel, and Fiona R. Kolbinger. The dresden surgical anatomy dataset for abdominal organ 16 segmentation in surgical data science. Scientific Data, 10(...

    2023

  3. [11]

    R.J. Chen, T. Ding, M.Y. Lu, et al. Towards a general- purpose foundation model for computational pathol- ogy. Nature Medicine, 30(5):850–862, 2024. 2

    2024

  4. [12]

    Expanding perfor- mance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding perfor- mance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024. 9

    2024 arXiv

  5. [13]

    Med-gemma: Medical vision-language models from google deepmind

    DeepMind. Med-gemma: Medical vision-language models from google deepmind. https : / / deepmind . google / models / gemma / medgemma/,

  6. [14]

    Multimodal whole slide foundation model for pathology

    Tong Ding, Sophia J Wagner, Andrew H Song, Richard J Chen, Ming Y Lu, Andrew Zhang, Anurag J Vaidya, Guillaume Jaume, Muhammad Sha- ban, Ahrong Kim, et al. Multimodal whole slide foundation model for pathology. arXiv preprint arXiv:2411.19666, 2024. 2

    2024 arXiv

  7. [15]

    Llm-assisted multi-teacher continual learning for visual question an- swering in robotic surgery, 2024

    Yuyang Du, Kexin Chen, Yue Zhan, Chang Han Low, Tao You, Mobarakol Islam, Ziyu Guo, Yueming Jin, Guangyong Chen, and Pheng-Ann Heng. Llm-assisted multi-teacher continual learning for visual question an- swering in robotic surgery, 2024. 23, 24

    2024

  8. [16]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023. 9, 15

    2023 arXiv

  9. [17]

    Cataract-1k dataset for deep-learning-assisted analysis of cataract surgery videos

    Nima Ghamsarian, Yehia El-Shabrawi, Saeed Nasiri- haghighi, Amin Madani, Pedro Marques, Maxime Gi- rard, Seyed Sadegh Mohseni Salehi, Jay Chien, Jean- Marc Mari, Te-Yuan Lee, and Alexandros Karargyris. Cataract-1k dataset for deep-learning-assisted analysis of cataract surgery...

  10. [18]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 9

    2025 arXiv

  11. [19]

    Khan, Sophia Bano, Hani J

    Runlong He, Mengya Xu, Adrito Das, Danyal Z. Khan, Sophia Bano, Hani J. Marcus, Danail Stoyanov, Matthew J. Clarkson, and Mobarakol Islam. Pitvqa: Image-grounded text embedding llm for visual ques- tion answering in pituitary surgery, 2024. 4, 23, 24, 27

    2024

  12. [20]

    Lora: Low-rank adaptation of large lan- guage models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large lan- guage models. ICLR, 1(2):3, 2022. 7

    2022

  13. [21]

    Ophnet: A large-scale video benchmark for ophthalmic surgical workflow under- standing, 2024

    Ming Hu, Peng Xia, Lin Wang, Siyuan Yan, Feilong Tang, Zhongxing Xu, Yimin Luo, Kaimin Song, Jurgen Leitner, Xuelian Cheng, Jun Cheng, Chi Liu, Kaijing Zhou, and Zongyuan Ge. Ophnet: A large-scale video benchmark for ophthalmic surgical workflow under- standing, 2024. 4, 23, 24, 27

    2024

  14. [22]

    Gpt- 4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt- 4o system card. arXiv preprint arXiv:2410.21276 , 2024. 4, 7, 9, 11, 12

    2024 arXiv

  15. [23]

    Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vi- sion–language representation learning with noisy text supervision. In Proceedings of the 38th International Con- ference on Ma...

  16. [24]

    Surgical visual question answering: A new frontier for interpretable computer-assisted intervention

    Adri ´an Jim´enez-S´anchez, Sebastian Bodenstedt, Jørgen Pedersen, Stefanie Speidel, Francisco Miguel S ´anchez- Margallo, and Jos ´e Garc´ıa-Rodr´ıguez. Surgical visual question answering: A new frontier for interpretable computer-assisted intervention. In International Confe...

    2022

  17. [25]

    Segcol challenge: Semantic segmentation for tools and fold edges in colonoscopy data, 2024

    Xinwei Ju, Rema Daher, Razvan Caramalau, Baoru Huang, Danail Stoyanov, and Francisco Vasconcelos. Segcol challenge: Semantic segmentation for tools and fold edges in colonoscopy data, 2024. 23, 24, 27

    2024

  18. [26]

    Lavanchy, Sanat Ramesh, Diego Dall’Alba, Cris- tians Gonzalez, Paolo Fiorini, Beat P

    Joel L. Lavanchy, Sanat Ramesh, Diego Dall’Alba, Cris- tians Gonzalez, Paolo Fiorini, Beat P . Muller-Stich, Philipp C. Nett, Jacques Marescaux, Didier Mutter, and Nicolas Padoy. Challenges in multi-centric generaliza- tion: Phase and step recognition in roux-en-y gastric bypa...

  19. [27]

    Llava- med: Training a large language-and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava- med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541...

    2023

  20. [28]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 20036–20046, 202...

    2023

  21. [29]

    Baichuan-omni-1.5 technical re- port

    Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. Baichuan-omni-1.5 technical re- port. arXiv preprint arXiv:2501.15368, 2025. 4

    2025

  22. [30]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 4, 9, 11, 12

    2023

  23. [31]

    Surgical sam 2: Real-time segment anything in surgical video by efficient frame pruning

    Haofeng Liu, Erli Zhang, Junde Wu, Mingxuan Hong, and Yueming Jin. Surgical sam 2: Real-time segment anything in surgical video by efficient frame pruning. arXiv preprint arXiv:2408.07931, 2024. 4

    2024 arXiv

  24. [32]

    Expert-level vision-language foundation model for real-world radiology and com- prehensive evaluation

    Xiaohong Liu, Guoxing Yang, Yulin Luo, Jiaji Mao, Xi- ang Zhang, Ming Gao, Shanghang Zhang, Jun Shen, 17 and Guangyu Wang. Expert-level vision-language foundation model for real-world radiology and com- prehensive evaluation. arXiv preprint arXiv:2409.16183,

    arXiv

  25. [33]

    Radiology- llama2: Best-in-class large language model for radiol- ogy, 2023

    Zhengliang Liu, Yiwei Li, Peng Shu, Aoxiao Zhong, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Jie Luo, Cheng Chen, Sekeun Kim, Jiang Hu, Haixing Dai, Lin Zhao, Dajiang Zhu, Jun Liu, Wei Liu, Dinggang Shen, Tianming Liu, Quanzheng Li, and Xiang Li. Radiology- llama2: Best-in-cla...

    2023

  26. [34]

    Surgraw: Multi-agent workflow with chain- of-thought reasoning for surgical intelligence

    Chang Han Low, Ziyue Wang, Tianyi Zhang, Zhitao Zeng, Zhu Zhuo, Evangelos B Mazomenos, and Yuem- ing Jin. Surgraw: Multi-agent workflow with chain- of-thought reasoning for surgical intelligence. arXiv preprint arXiv:2503.10265, 2025. 4, 23, 24, 28

    2025

  27. [35]

    A visual-language foundation model for com- putational pathology

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foundation model for com- putational pathology. Nature Medicine, 30(3):863–874,

  28. [36]

    A multimodal generative ai copilot for human pathology

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Melissa Zhao, Aaron K Chow, Kenji Ikemura, Ahrong Kim, Dimitra Pouli, Ankush Patel, et al. A multimodal generative ai copilot for human pathology. Nature, 634(8033):466–473, 2024. 2

    2024

  29. [37]

    Nunez Do Rio, Lyn- don da Cruz, Christos Bergeles, Hongyu Chen, Fu- cang Jia, Nikhil KumarTomar, Debesh Jha, Michael A

    Imanol Luengo, Maria Grammatikopoulou, Rahim Mohammadi, Chris Walsh, Chinedu Innocent Nwoye, Deepak Alapatt, Nicolas Padoy, Zhen-Liang Ni, Chen- Chen Fan, Gui-Bin Bian, Zeng-Guang Hou, Heonjin Ha, Jiacheng Wang, Haojie Wang, Dong Guo, Lu Wang, Guotai Wang, Mobarakol Islam, Bha...

    2020

  30. [38]

    Endoscapes2023, a critical view of safety and surgical scene segmentation dataset for laparoscopic cholecys- tectomy, 2024

    Pietro Mascagni, Deepak Alapatt, Aditya Murali, Armine Vardazaryan, Alain Garcia Vazquez, Nariaki Okamoto, Guido Costamagna, Didier Mutter, Jacques Marescaux, Bernard Dallemagne, and Nicolas Padoy. Endoscapes2023, a critical view of safety and surgical scene segmentation datas...

    2024

  31. [39]

    National institutes of health

    National Institutes of Health (NIH). National institutes of health. https://www.nih.gov/, 2025. 5

    2025

  32. [40]

    Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos

    Chinedu Innocent Nwoye, Tong Yu, Cristians Gonza- lez, Barbara Seeliger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Medical Image Analysis, 78:1024...

    2022

  33. [41]

    Cholectrack20: A multi- perspective tracking dataset for surgical tools

    Chinedu Innocent Nwoye, Fauzan Zaid, Jo ¨el La- vanchy, and Nicolas Padoy. Cholectrack20: A multi- perspective tracking dataset for surgical tools. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Wed June 11th-Sun June 15th, 2025, 2025. 4

    2025

  34. [42]

    Foundation models in radiology: What, how, why, and why not

    Magdalini Paschali, Zhihong Chen, Louis Blankemeier, Maya Varma, Alaa Youssef, Christian Bluethgen, Cur- tis Langlotz, Sergios Gatidis, and Akshay Chaudhari. Foundation models in radiology: What, how, why, and why not. Radiology, 314(2):e240597, 2025. 2

    2025

  35. [43]

    Skywork r1v: Pioneering multi- modal reasoning with chain-of-thought

    Yi Peng, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Wei- jie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyi- dan Xie, Li Ge, et al. Skywork r1v: Pioneering multi- modal reasoning with chain-of-thought. arXiv preprint arXiv:2504.05599, 2025. 9, 11, 12

    2025 arXiv

  36. [44]

    Competence-based curriculum learning for neural ma- chine translation

    Triantafyllos Platanios, Eric Gibson, and Tom Mitchell. Competence-based curriculum learning for neural ma- chine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1162–1172, 2020. 15

    2020

  37. [45]

    Sar-rarp50: Segmentation of surgical in- strumentation and action recognition on robot-assisted radical prostatectomy challenge, 2024

    Dimitrios Psychogyios, Emanuele Colleoni, Beat- rice Van Amsterdam, Chih-Yang Li, Shu-Yu Huang, Yuchong Li, Fucang Jia, Baosheng Zou, Guotai Wang, Yang Liu, Maxence Boels, Jiayu Huo, Rachel Sparks, Prokar Dasgupta, Alejandro Granados, Sebastien Ourselin, Mengya Xu, An Wang, Ya...

    2024

  38. [46]

    Learning transferable visual models from natural lan- guage supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. InInternational conference on machine learning,...

    2021

  39. [47]

    Rios, M.A

    M.S. Rios, M.A. Molina-Rodriguez, D. Londono, and et al. Cholec80-cvs: An open dataset with an evalua- tion of strasberg’s critical view of safety for ai.Scientific Data, 10(1):194, 2023. 23, 24, 25

    2023

  40. [48]

    Surgical-VQA: Visual question answering in surgical scenes using trans- former

    Lalithkumar Seenivasan, Mobarakol Islam, Adithya K Krishna, and Hongliang Ren. Surgical-VQA: Visual question answering in surgical scenes using trans- former. In Proceedings of the 25th International Conference on Medical Image Computing and Computer-Assisted Inter- vention (M...

    2022

  41. [49]

    Surgicalgpt: end-to-end language-vision gpt for visual question answering in surgery

    Lalithkumar Seenivasan, Mobarakol Islam, Gokul Kan- nan, and Hongliang Ren. Surgicalgpt: end-to-end language-vision gpt for visual question answering in surgery. In International conference on medical image com- 18 puting and computer-assisted intervention, pages 281–290. Spri...

    2023

  42. [50]

    Think step by step: Chain-of-gesture prompting for error detection in robotic surgical videos

    Zhimin Shao, Jialang Xu, Danail Stoyanov, Evange- los B Mazomenos, and Yueming Jin. Think step by step: Chain-of-gesture prompting for error detection in robotic surgical videos. IEEE Robotics and Automation Letters, 2024. 4

    2024

  43. [51]

    Gemini: a family of highly capable multi- modal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Milli- can, et al. Gemini: a family of highly capable multi- modal models. arXiv preprint arXiv:2312.11805, 2023. 4, 7, 9, 11, 12

    2023 arXiv

  44. [52]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram ´e, Mor- gane Rivi `ere, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. 9, 11, 12

    2025 arXiv

  45. [53]

    Kimi-vl technical re- port

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical re- port. arXiv preprint arXiv:2504.07491, 2025. 4

    2025 arXiv

  46. [54]

    Minicpm-o 2.6: A gpt- 4o level mllm for vision, speech, and multimodal live streaming on your phone, 2025

    OpenBMB MiniCPM-o Team. Minicpm-o 2.6: A gpt- 4o level mllm for vision, speech, and multimodal live streaming on your phone, 2025. 9, 11, 12

    2025

  47. [55]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 9

    2023 arXiv

  48. [56]

    Siglip 2: Multilingual vision-language encoders with improved semantic un- derstanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic un- derstanding, localization, an...

    2025 arXiv

  49. [57]

    Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy

    Apurva B. Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. Endonet: A deep architecture for recognition tasks on laparoscopic videos. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4121–4129. IEEE,...

    2016

  50. [58]

    Molecular-driven foundation model for oncologic pathology

    Anurag Vaidya, Andrew Zhang, Guillaume Jaume, Andrew H Song, Tong Ding, Sophia J Wagner, Ming Y Lu, Paul Doucet, Harry Robertson, Cristina Almagro- Perez, et al. Molecular-driven foundation model for oncologic pathology. arXiv preprint arXiv:2501.16652 ,

    arXiv

  51. [59]

    A foundation model for clinical- grade computational pathology and rare cancers detec- tion

    Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Kristen Sev- erson, Eric Zimmermann, James Hall, Neil Tenenholtz, Nicolo Fusi, et al. A foundation model for clinical- grade computational pathology and rare cancers detec- tion. Nature medic...

    2024

  52. [60]

    Copesd: A multi-level surgical motion dataset for training large vision-language models to co- pilot endoscopic submucosal dissection, 2024

    Guankun Wang, Han Xiao, Huxin Gao, Renrui Zhang, Long Bai, Xiaoxiao Yang, Zhen Li, Hongsheng Li, and Hongliang Ren. Copesd: A multi-level surgical motion dataset for training large vision-language models to co- pilot endoscopic submucosal dissection, 2024. 2, 4, 23, 24, 25

    2024

  53. [61]

    Endochat: Grounded multimodal large language model for endoscopic surgery

    Guankun Wang, Long Bai, Junyi Wang, Kun Yuan, Zhen Li, Tianxu Jiang, Xiting He, Jinlin Wu, Zhen Chen, Zhen Lei, et al. Endochat: Grounded multimodal large language model for endoscopic surgery. arXiv preprint arXiv:2501.11347, 2025. 2

    2025 arXiv

  54. [62]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any reso- lution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model’s perception of the world at any reso- lution. arXiv preprint arXiv:2409.12191, 2024. 4

    2024 arXiv

  55. [63]

    A pathology foundation model for cancer diagnosis and prognosis prediction

    Xiyue Wang, Junhan Zhao, Eliana Marostica, Wei Yuan, Jietian Jin, Jiayu Zhang, Ruijiang Li, Hongping Tang, Kanran Wang, Yu Li, et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature, 634(8035):970–978, 2024. 2

    2024

  56. [64]

    Auto- laparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hys- terectomy, 2022

    Ziyi Wang, Bo Lu, Yonghao Long, Fangxun Zhong, Tak-Hong Cheung, Qi Dou, and Yunhui Liu. Auto- laparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hys- terectomy, 2022. 4, 23, 24

    2022

  57. [65]

    Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. arXiv preprint arXiv:2308.02463, 2023. 2

    2023 arXiv

  58. [66]

    Deepseek- vl2: Mixture-of-experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek- vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024. 4

    2024 arXiv

  59. [67]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. 9

    2024 arXiv

  60. [68]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 4

    2024 arXiv

  61. [69]

    mplug-owl3: Towards long image-sequence under- standing in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence under- standing in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024. 4

    2024 arXiv

  62. [70]

    Learning multi-modal representations by watching hundreds of surgical video lectures

    Kun Yuan, Vinkle Srivastav, Tong Yu, Joel L La- vanchy, Pietro Mascagni, Nassir Navab, and Nico- las Padoy. Learning multi-modal representations by watching hundreds of surgical video lectures. arXiv preprint arXiv:2307.15220, 2023. 2

    2023 arXiv

  63. [71]

    Advanc- ing surgical vqa with scene graph knowledge

    Kun Yuan, Manasi Kattel, Jo ¨el L Lavanchy, Nassir Navab, Vinkle Srivastav, and Nicolas Padoy. Advanc- ing surgical vqa with scene graph knowledge. Interna- tional Journal of Computer Assisted Radiology and Surgery, 19(7):1409–1417, 2024. 2 19

    2024

  64. [72]

    Cognition guided human-object re- lationship detection

    Zhitao Zeng, Pengwen Dai, Xuan Zhang, Lei Zhang, and Xiaochun Cao. Cognition guided human-object re- lationship detection. IEEE Transactions on Image Process- ing, 32:2468–2480, 2023. 4

    2023

  65. [73]

    Sigmoid loss for language image pre- training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF international conference on computer vision , pages 11975–11986, 2023. 9

    2023

  66. [74]

    Internlm-xcomposer2

    Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al. Internlm-xcomposer2. 5- omnilive: A comprehensive multimodal system for long-term streaming video and audio interactions. arXiv preprint arXiv:2412.09596...

    2024 arXiv

  67. [75]

    Long con- text transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Hao- ran Tan, Chunyuan Li, and Ziwei Liu. Long con- text transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 4

    2024 arXiv

  68. [76]

    Knowledge-enhanced visual-language pre-training on chest radiology images.Nature Commu- nications, 14(1):4542, 2023

    Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanced visual-language pre-training on chest radiology images.Nature Commu- nications, 14(1):4542, 2023. 2

    2023

  69. [77]

    Large-scale long-tailed disease diagnosis on radiology images

    Qiaoyu Zheng, Weike Zhao, Chaoyi Wu, Xiaoman Zhang, Lisong Dai, Hengyu Guan, Yuehua Li, Ya Zhang, Yanfeng Wang, and Weidi Xie. Large-scale long-tailed disease diagnosis on radiology images. Na- ture Communications, 15(1):10147, 2024. 2

    2024

  70. [78]

    Internvl3: Exploring advanced training and test-time recipes for open-source multi- modal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Wei- jie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multi- modal models. arXiv preprint arXiv:2504.10479 , 2025. 9, 11, 12, 13...

    2025 arXiv

  71. [79]

    The demonstrated step is gastrojejunal defect closure, and during the step, the surgeon is closing the orifice left by the stapler, creating the gastrojejunostomy

    The demonstrated phase is gastrojejunal anastomosis, during the phase, the surgeon is performing an anastomosis of the small bowel with the gastric pouch. The demonstrated step is gastrojejunal defect closure, and during the step, the surgeon is closing the orifice left by the...

  72. [80]

    Surgical Instruments : Metal tools, such as forceps or scissors, are visible, indicating precise manipulation of tissue

  73. [81]

    Tissue and Organs : The surrounding area includes soft tissue and what appears to be part of the gastrointestinal tract or abdominal cavity

  74. [82]

    LLM Decoder V V V V V V TTTT Multimodal Fusion Vision Encoder Text Tokenizer �� �푀 �� �� 1

    Stitching or Suturing : A suture thread is visible, suggesting that suturing or repair work is being conducted. LLM Decoder V V V V V V TTTT Multimodal Fusion Vision Encoder Text Tokenizer �� �푀 �� �� 1. What phase and step does this gastric bypass procedure image or video depict?

  75. [83]

    What instrument used in this surgical scene?

  76. [84]

    Describe the categories of tissue or organ shown in surgical image

  77. [85]

    Overall illustration of proposed SurgVLM

    What is the action of surgeon in this timestamp? Dynamic Resolution Unified Autoregressive Loss for 10 Surgical Tasks Dynamic Fine-tuning Strategy Figure S1. Overall illustration of proposed SurgVLM. SurgVLM model follows the Qwen2.5-VL architecture, consisting of a vision enc...

  78. [86]

    Visual Perception: Instrument Recognition; Instrument Localization with Box; Instrument Localization with Grid; Tissue Recognition; Tissue Localization

  79. [87]

    Temporal Analysis: Phase Recognition; Step Recognition; Action Recognition; Triplet Recognition

  80. [88]

    present” vs. “absent

    Reasoning: Critical View of Safety Assessment. This hierarchical structuring ensures that the model learns correlations between tasks, effectively interpreting comprehensive multi-task scene descriptions to enhance single-task inference capabilities. 22 Intra-task Hierarchical...

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.