Pith. sign in

REVIEW 4 major objections 5 minor 45 references

SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Surgical video understanding can be done by retrieving answers from a surgery vocabulary instead of generating them token by token.

desk verdict SurgNarrator is a credible applied paper, but the central retrieval mechanism is only tested on an in-house benchmark that shares data with the vocabulary; the external zero-shot results validate the fine-tuned embedding, not the generative retrieval itself. read the letter →

arxiv 2608.04676 v1 pith:XIDXHQF4 submitted 2026-08-05 cs.CV

Yuqing Feng , Jiawei Ma , Kevin Qinghong Lin , Kun Yuan , Nicolas Padoy , Daniel S. Elson , Anh Nguyen , Stamatia Giannarou
show 1 more author
Baoru Huang
This is my paper · ORCID
classification cs.CV
keywords surgicalvideounderstandinggenerativeretrievalcontrastivelearningvideo-languagemodelzero-shottemporalreasoningsurgery-centricvocabularyprocedure-aware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SurgNarrator tries to close the gap between flexible but slow generative video-language models and fast but shallow contrastive matchers in surgery: it turns surgical video understanding into query-conditioned retrieval from a compact vocabulary of clinical concepts. The paper argues that a surgery-centric vocabulary, built by extracting action, instrument, and intent phrases from surgical captions, can serve as the answer space for recognition and reasoning tasks alike. A fine-tuned multimodal embedding model learns temporally discriminative representations, and a hierarchical procedure-aware retrieval step narrows the search. If the claims hold, surgical video understanding can run at retrieval-level latency while supporting temporal and intent questions, not just static classification.

What carries the argument

The central object is the surgery-centric vocabulary: more than 31K verb–noun action phrases, instrument names, and clinical intent phrases, organized by procedure type, extracted from SurgLaVi-β captions by Qwen3-32B and normalized. It is both the answer space and the semantic substrate for every task. The second mechanism is a temporally-aware contrastive objective built on InfoNCE: temporally adjacent clips from the same video and semantic level are mined as hard negatives, and candidates whose answer embeddings are highly similar to the positive answer are masked as false negatives at a similarity threshold of 0.80. The third is hierarchical procedure-aware retrieval, which first retrieves a procedure type by cosine similarity to cached procedure embeddings and then retrieves the final answer only from that procedure's vocabulary subset.

What would settle it

Take any ground-truth answer from Surg-Eval or the downstream benchmarks and check whether it appears in the surgery-centric vocabulary; the fraction of missing answers is the ceiling on Recall@K. A direct test would remove an answer from the candidate set and confirm that Recall@K drops to zero, or equivalently measure coverage of all ground-truth labels across the twelve benchmarks.

Watch

Extended reading notes

Core claim

The paper's central claim is that generative retrieval can be made to work for surgery, and that it beats both autoregressive and contrastive alternatives on the paper's own evaluation. SurgNarrator outperforms a fine-tuned Qwen3-VL-8B-Instruct and a strong contrastive baseline, SurgCLIP-β, on instrument and action recognition, temporal perception, and intent reasoning, while reducing output-stage latency by roughly two orders of magnitude compared with the generative baseline. In zero-shot transfer to twelve downstream benchmarks, it improves over prior surgical vision-language models on most phase, step, action, triplet, and tool recognition tasks, despite being trained only on an 86.6% subset of the data used by the strongest contrastive baseline. The paper attributes these gains to grounding retrieval in a caption-extracted surgery vocabulary, a temporally-aware contrastive objective with false-negative masking, and restricting the final retrieval space to the top-1 procedure type.

Load-bearing premise

The load-bearing premise is that the caption-extracted vocabulary is complete enough: any correct answer that is not one of the 31K-plus entries cannot be returned, because retrieval cannot produce it.

Editorial extensions

If this is right

  • Surgical video understanding can support temporal and intent questions at retrieval-level latency, making real-time intraoperative assistance more plausible.
  • Restricting the answer space to a procedure-specific subset improves accuracy over full-vocabulary retrieval, so procedure priors act as an effective clinical constraint.
  • Training with temporal hard negatives and false-negative masking transfers to phase, step, action, and tool recognition, with consistent reported gains over standard InfoNCE.
  • The same model can handle both recognition and reasoning tasks without changing the output head, because both are expressed as retrieval from one vocabulary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit is that the method's ceiling is set by vocabulary coverage: if a correct answer is absent from the 31K-plus entries, Recall@K for that query is zero by construction.
  • A testable next step is measuring the fraction of ground-truth answers across Surg-Eval and the twelve downstream benchmarks that actually appear in the vocabulary, and then adding missing answers to see how much accuracy rises.
  • Because outputs are restricted to caption-grounded clinical phrases, retrieval may act as a hallucination guardrail, but the same constraint makes the system fragile when the correct answer is rare in the caption data used to build the vocabulary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SurgNarrator, a generative retrieval framework for surgical video understanding. It constructs a surgery-centric vocabulary from SurgLaVi-β captions using Qwen3-32B, fine-tunes Qwen3-VL-Embedding-8B with a temporally-aware contrastive objective that includes temporal hard-negative mining and false-negative masking, and at inference performs hierarchical procedure-aware retrieval: first a procedure type, then an answer from the procedure-specific vocabulary. The authors evaluate on an in-house Surg-Eval benchmark and on twelve external zero-shot benchmarks, reporting gains over generative and contrastive baselines and a reduction in output-stage latency by two orders of magnitude. The central claim is that surgical video understanding, including temporal and intent reasoning, can be performed accurately at retrieval-level latency by retrieving from a surgery-centric vocabulary instead of autoregressively generating text.

Significance. If fully supported, the work would be a useful step toward low-latency surgical video understanding. The idea of grounding retrieval in a structured clinical concept space is compelling and differentiates the method from generic contrastive retrieval. The temporal hard-negative mining with false-negative masking is a sensible adaptation to the near-duplicate visual appearance of adjacent surgical clips. The twelve external benchmark results, if taken at face value, would also indicate strong transferable representations from the fine-tuned backbone. However, the evaluation architecture has two gaps: the external benchmarks do not exercise the proposed retrieval mechanism, and the in-house benchmark has an unmeasured vocabulary-coverage ceiling and a shared provenance with the vocabulary. The significance is therefore conditional on additional analysis and clarification.

major comments (4)
  1. [Section IV-C and Tables III–V] The twelve zero-shot benchmarks do not evaluate the generative-retrieval mechanism that is the paper's central contribution. The protocol in Sec. IV-C embeds each 16-frame window and compares it with candidate class embeddings using cosine similarity; there is no surgery-centric vocabulary and no hierarchical procedure-aware retrieval (Section III-E). Consequently, the consistent gains in Tables III–V support the quality of the fine-tuned Qwen3-VL-Embedding backbone, not the vocabulary or the Eq. (14) retrieval. The abstract's claim that the method is comprehensively evaluated on twelve benchmarks therefore overstates what these experiments establish.
  2. [Section III-C and Section IV-A2] The paper never specifies whether the Qwen3-32B vocabulary extraction is restricted to the training split of SurgLaVi-β or includes the held-out evaluation split. Since Surg-Eval is built from the same SurgLaVi-β source, inclusion of held-out captions would make the ground-truth answers trivially present in the vocabulary, while exclusion would leave coverage unmeasured. Either way, the Recall@K values in Tables I, II, and VI are upper-bounded by an unknown completeness property: for any ground-truth answer absent from A_{\hat p_i}, Eq. (14) cannot retrieve it, so Recall@K is zero by construction. The paper should report the fraction of Surg-Eval ground-truth answers covered by the full vocabulary A and by each procedure-specific subset A_{\hat p_i}, an oracle ceiling with perfect procedure retrieval, and a faithfulness check of the Qwen3-32B extraction, for example against SurgLaVi-β's hierarchical annotations.
  3. [Section IV-F, Fig. 5, Tables III–V] The false-negative masking threshold δ=0.80 is selected by ablating on the same downstream benchmarks (Fig. 5) on which the final zero-shot results are then reported. This is a selection-on-test procedure; the reported gains of 4.52 percentage points on phase/step/action and 3.66 mAP points on tool recognition are optimistically biased. The threshold should be fixed on a validation split (for example, a held-out portion of Surg-Train) and only then applied to the external benchmarks, or the paper should report results for all thresholds with a clear statement of the selection protocol.
  4. [Section III-E and Table VI] The hierarchical retrieval ablation is informative, but it is performed only on Surg-Eval, where the vocabulary-coverage issue described above directly affects the denominator of all Recall@1 numbers. If, for example, the top-1 procedure subset frequently lacks the ground-truth answer while a top-3 union contains it, the ranking of strategies in Table VI could change. The authors should repeat Table VI after reporting coverage and oracle ceilings, and should also report how often the correct procedure type is retrieved in Eq. (12), since a wrong procedure type makes the final answer unreachable in principle.
minor comments (5)
  1. [Section IV-A1] The phrase 'original SurgCLIP-β [31] pretraining data' should read 'original SurgLaVi-β pretraining data' (or be clarified), since SurgCLIP-β is a model, not a dataset.
  2. [Tables I and II] The reported decode time for SurgNarrator appears to include only the final vocabulary lookup in some rows; please clarify whether the hierarchical procedure-retrieval stage (Section III-E) is included in the reported per-query decode time, given that Fig. 6 reports separate decode times for the hierarchical setting.
  3. [Fig. 5] The bar chart does not show error bars or repeated-run variance; given the small absolute differences between some thresholds (e.g., δ=0.85 versus δ=0.80 on several tasks), the selection of δ=0.80 should be accompanied by variance estimates.
  4. [Section III-A] The notation A_p is introduced but the set of procedure types P is only implicit; please state explicitly that P is the set of procedures covered by the vocabulary.
  5. [Global] There are several typographical and formatting issues, including 'AUTHORet al.' in the running header, broken author initials such as 'V . Srivastav' in the references, and inconsistent use of unicode math symbols; these should be cleaned in the final version.

Circularity Check

1 steps flagged · score 4.0 of 10

Surg-Eval's candidate vocabulary is built from the same SurgLaVi-β captions that define its ground-truth answers; the twelve external benchmarks exercise only the fine-tuned embedding, so the core generative-retrieval claim lacks fully independent validation.

  1. self definitional [Section III-C (Surgery-Centric Vocabulary Construction) and Section IV-A2 (Evaluation Datasets)]
    "The vocabulary is constructed from SurgLaVi-β [31], which provides surgical video clips paired with descriptive captions and hierarchical annotations. ... Surg-Eval is constructed from the evaluation split of SurgLaVi-β [31], with no overlap with the training videos."

    The surgery-centric vocabulary A is extracted by Qwen3-32B from SurgLaVi-β captions, and the Surg-Eval ground-truth answers are also drawn from the same SurgLaVi-β caption/annotation hierarchy. The paper does not state that evaluation-split captions were withheld from vocabulary construction; indeed, it says 'Each caption is decomposed into three semantic categories.' Equation (14) restricts the answer space to A_p̂, so any answer absent from A cannot be retrieved. If evaluation captions are included, the candidate set is defined by the same text that defines the labels, making Recall@K a self-referential measurement: it cannot detect whether the vocabulary covers unseen clinically meaningful answers, and no coverage statistics are reported.

full rationale

The paper's central contribution is the surgery-centric vocabulary plus hierarchical procedure-aware retrieval (Eqs. 11-14). The only experiments that actually use this mechanism are on Surg-Eval (Tables I-II, VI). Surg-Eval is built from the held-out split of SurgLaVi-β, while the vocabulary is constructed from SurgLaVi-β captions via Qwen3-32B; the paper does not state that evaluation-split captions were excluded from vocabulary construction. Since Eq. (14) restricts the answer space to A_p̂, the reported Recall@K is measured against a candidate set that was derived from the same corpus that defines the ground-truth labels. This is a self-definitional evaluation: it cannot detect whether the vocabulary covers clinically meaningful answers outside the training captions, and no coverage statistics are reported. This warrants a partial circularity score of 4. The twelve zero-shot benchmarks (Tables III-V) are external and independent, but per Section IV-C they use only the 16-frame-window embedding compared with candidate class embeddings by cosine similarity, without the vocabulary or Eq. (14). Those gains therefore validate the fine-tuned Qwen3-VL-Embedding backbone rather than the generative-retrieval mechanism. The self-citations to VLog, SurgVLP, HecVL, and PeskaVLP are used as prior-work context and baselines, not as load-bearing justification, so they do not increase the score. Overall, partial circularity exists in the in-house Surg-Eval evaluation, while the external benchmarks provide independent but narrower support for the embedding component.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two families of assumptions: the completeness and faithfulness of the LLM-extracted surgery vocabulary, and the validity of temporal adjacency as a hard-negative signal. The numerical free parameters are standard hyperparameters except for the threshold delta, which is the most consequential because it was chosen on evaluation benchmarks. No invented entities are introduced.

free parameters (3)
  • False-negative masking threshold delta = 0.80
    Selected by ablating on the evaluation benchmarks (Fig. 5) and then used for all reported results. It controls the loss in Eq. (8)-(10) and is a consequential design choice.
  • Temporal neighborhood radius r = 1
    Manually set to select immediately preceding and following clips as hard negatives in Eq. (6). No sensitivity analysis is reported for this parameter.
  • Contrastive temperature gamma = 0.07
    A common InfoNCE temperature set in Section IV-C. It scales the loss in Eq. (5) and Eq. (10), but is not the main focus of the method.
assumptions (4)
  • domain assumption Surgical workflows are structured and recurring, so a procedure-organized vocabulary is a valid output space for understanding.
    Stated in the introduction and used throughout Section III-C. If surgical workflows were not structured, the procedure-specific retrieval space would not be natural.
  • ad hoc to paper Qwen3-32B caption extraction is faithful and complete, producing all clinically relevant actions, instruments, and intents from the captions.
    Section III-C relies on this LLM to build the vocabulary. No quantitative coverage or faithfulness validation is provided.
  • domain assumption Temporally adjacent clips at the same semantic level with |tau_j - tau_i| <= 1 are semantically distinct enough to serve as hard negatives.
    The temporal hard-negative set in Eq. (6) assumes neighboring clips are informative negatives. The masking threshold partially mitigates false negatives, but the assumption remains load-bearing.
  • domain assumption The top-1 retrieved procedure type is correct often enough that restricting the retrieval space to A_p does not exclude the correct answer.
    Eq. (12)-(14) rely on this. If the procedure type is misclassified, the correct answer is outside the retrieval space by construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding." pith.science (2026). https://pith.science/paper/XIDXHQF4

@misc{pith2026260804676,
  author       = {Pith},
  title        = {Pith review of: SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XIDXHQF4}},
  note         = {Machine review of arXiv:2608.04676}
}
read the original abstract

Surgical procedures unfold as structured and recurring clinical events, whose real-time understanding via intraoperative surgical videos is critical for intraoperative decision-making and support. However, existing video understanding methods force a trade-off: autoregressive video-language models support comprehensive reasoning but are not practical for time-sensitive clinical applications, whereas contrastive models offer low latency but struggle with complex scene understanding. Recently, generative retrieval has been explored for general-domain video understanding, but transferring it to surgery is not trivial because near-identical visual appearances may indicate semantically distinct events, and the terminology involved is highly surgery-specific. To this end, we propose SurgNarrator, a new generative retrieval framework tailored for surgical video understanding. We construct a well-curated surgery-centric vocabulary from surgical captions to define a clinically meaningful retrieval space. We then adapt the pre-trained Qwen3-VL-Embedding-8B to learn discriminative clinical representations with a temporally-aware contrastive objective. During inference, a hierarchical, procedure-aware retrieval strategy narrows the search space to the relevant procedure type, delivering fast and effective responses. Our method is comprehensively evaluated on twelve benchmarks in a zero-shot setting and achieves consistent performance gains over state-of-the-art baselines, while reducing output-stage latency by more than two orders of magnitude compared with the generative baseline.

Figures

Figures reproduced from arXiv: 2608.04676 by the authors.

Figure 1
Figure 1. Overview of SurgNarrator. SurgNarrator adapts generative retrieval to surgical video understanding by grounding retrieval in a surgery￾centric vocabulary of clinical concepts. The vocabulary is organized by procedure type and semantic category, supporting tasks such as action recognition, instrument recognition, temporal perception, and intent reasoning without token-by-token autoregressive decoding. discriminative … view at source ↗
Figure 2
Figure 2. SurgNarrator reformulates surgical video understand [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 2
Figure 2. Hierarchical procedure-aware generative retrieval pipeline of SurgNarrator. Vocabulary and procedure-type embeddings are pre-computed and cached for efficient inference. Given a surgical video-query pair, SurgNarrator obtains a dense video-query representation from the last hidden state of the appended PAD token. It first identifies the procedure type, then retrieves the answer from the corresponding procedure-speci… view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: The process of surgery-centric vocabulary construction. We employ Qwen3-32B to extract raw vocabulary entries from the captions along three dimensions: action, instrument, and intent. After careful filtering and deduplication, the resulting terms are organized by proce…
Figure 4
Figure 4. Figure 4: Temporally adjacent surgical clips share a nearly identical visual appearance while corresponding to semantically distinct surgical events. D. Temporally-Aware Contrastive Learning Standard contrastive learning aligns matched video-text pairs by treating non-matching s…
Figure 5
Figure 5. Figure 5: Ablation study of temporally-aware hard negative sampling. We compare standard InfoNCE, temporally adjacent hard negatives without false-negative masking (w/ Negatives), and different similarity thresholds for false-negative masking. The threshold δ = 0.80 provides the…
Figure 6
Figure 6. Figure 6: Effect of the procedure-type retrieval space on retrieval accuracy and inference efficiency. Enlarging the procedure-type retrieval space consistently reduces Recall@1 while increasing decoding time across all tasks. may contain semantically overlapping candidates that…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 29 canonical work pages

  1. [1]

    Surgical data science for next-generation interventions,

    L. Maier-Heinet al., “Surgical data science for next-generation interventions,”Nature Biomedical Engineering, vol. 1, no. 9, p. 691–696,

    work page

  2. [2]

    Endonet: a deep architecture for recognition tasks on laparoscopic videos,

    A. P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. De Mathelin, and N. Padoy, “Endonet: a deep architecture for recognition tasks on laparoscopic videos,”IEEE Trans. Med. Imag., vol. 36, no. 1, pp. 86– 97, 2016

    work page 2016

  3. [3]

    Tecno: Surgical phase recognition with multi-stage temporal convolutional networks,

    T. Czempielet al., “Tecno: Surgical phase recognition with multi-stage temporal convolutional networks,” inProc. MICCAI. Springer, 2020, pp. 343–352

    work page 2020

  4. [4]

    Video-based surgical skill assessment using 3d convolutional neural networks,

    I. Funke, S. T. Mees, J. Weitz, and S. Speidel, “Video-based surgical skill assessment using 3d convolutional neural networks,”Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 7, pp. 1217–1225, 2019

    work page 2019

  5. [5]

    CholecTriplet2021: A benchmark challenge for surgical action triplet recognition,

    C. I. Nwoyeet al., “CholecTriplet2021: A benchmark challenge for surgical action triplet recognition,”Med. Image Anal., vol. 86, p. 102803, 2023

    work page 2023

  6. [6]

    Video-chatgpt: Towards detailed video understanding via large vision and language models,

    M. Maaz, H. Rasheed, S. Khan, and F. Khan, “Video-chatgpt: Towards detailed video understanding via large vision and language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 12 585– 12 602

    2024

  7. [7]

    Video-llama: An instruction-tuned audio- visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio- visual language model for video understanding,” inProceedings of the 2023 conference on empirical methods in natural language processing: system demonstrations, 2023, pp. 543–553

    work page 2023

  8. [8]

    Llava-surg: towards multimodal surgical assistant via structured surgical video learning,

    J. Liet al., “Llava-surg: towards multimodal surgical assistant via structured surgical video learning,”arXiv preprint arXiv:2408.07981, 2024

    arXiv 2024

Show all 45 references
  1. [9]

    The llama 3 herd of models,

    A. Grattafioriet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [10]

    Qwen3 technical report,

    A. Yanget al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  3. [11]

    CLIP4Clip: An empirical study of clip for end to end video clip retrieval and captioning,

    H. Luoet al., “CLIP4Clip: An empirical study of clip for end to end video clip retrieval and captioning,”Neurocomputing, vol. 508, pp. 293–304, 2022. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S0925231222008876

    2022

  4. [12]

    Videoclip: Contrastive pre-training for zero-shot video- text understanding,

    H. Xuet al., “Videoclip: Contrastive pre-training for zero-shot video- text understanding,” inProceedings of the 2021 conference on empirical methods in natural language processing, 2021, pp. 6787–6800

    2021

  5. [13]

    Learning multi-modal representations by watching hundreds of surgical video lectures,

    K. Yuanet al., “Learning multi-modal representations by watching hundreds of surgical video lectures,”Med. Image Anal., vol. 105, p. 103644, 2025

    2025

  6. [14]

    Vlog: Video-language models by generative retrieval of narration vocabulary,

    K. Q. Lin and M. Z. Shou, “Vlog: Video-language models by generative retrieval of narration vocabulary,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 3218–3228

    2025

  7. [15]

    Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking,

    M. Liet al., “Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking,”arXiv preprint arXiv:2601.04720, 2026

    2026 arXiv

  8. [16]

    Hecvl: Hierarchical video-language pretraining for zero-shot surgical phase recognition,

    K. Yuan, V . Srivastav, N. Navab, and N. Padoy, “Hecvl: Hierarchical video-language pretraining for zero-shot surgical phase recognition,” in Proc. MICCAI. Springer, 2024, pp. 306–316

    2024

  9. [17]

    VidLPRO: A video- language pre-training framework for robotic and laparoscopic surgery,

    M. Honarmand, M. A. Jamal, and O. Mohareri, “VidLPRO: A video- language pre-training framework for robotic and laparoscopic surgery,” inAdvancements In Medical Foundation Models: Explainability, Robust- ness, Security, and Beyond, 2024

    2024

  10. [18]

    Ophclip: Hierarchical retrieval-augmented learning for ophthalmic surgical video-language pretraining,

    M. Huet al., “Ophclip: Hierarchical retrieval-augmented learning for ophthalmic surgical video-language pretraining,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 19 838–19 849

    2025

  11. [19]

    Procedure-aware surgical video-language pretraining with hierarchical knowledge aug- mentation,

    K. Yuan, V . Srivastav, N. Navab, and N. Padoy, “Procedure-aware surgical video-language pretraining with hierarchical knowledge aug- mentation,”Advances in Neural Information Processing Systems, vol. 37, pp. 122 952–122 983, 2024

    2024

  12. [20]

    CliPPER: Contextual video-language pretraining on long-form intraoperative surgical proce- dures for event recognition,

    F. Stilz, V . Srivastav, N. Navab, and N. Padoy, “CliPPER: Contextual video-language pretraining on long-form intraoperative surgical proce- dures for event recognition,”arXiv preprint arXiv:2603.24539, 2026

    2026

  13. [21]

    SurgicalGPT: end- to-end language-vision gpt for visual question answering in surgery,

    L. Seenivasan, M. Islam, G. Kannan, and H. Ren, “SurgicalGPT: end- to-end language-vision gpt for visual question answering in surgery,” in Proc. MICCAI. Springer, 2023, pp. 281–290

    2023

  14. [22]

    Gp-vls: A general-purpose vision language model for surgery,

    S. Schmidgall, J. Cho, C. Zakka, and W. Hiesinger, “Gp-vls: A general-purpose vision language model for surgery,”arXiv preprint arXiv:2407.19305, 2024

    2024 arXiv

  15. [23]

    Memory-augmented multimodal llms for surgical vqa via self-contained inquiry,

    W. Hou, Y . Cheng, K. Xu, Y . Hu, W. Li, and J. Liu, “Memory-augmented multimodal llms for surgical vqa via self-contained inquiry,”arXiv preprint arXiv:2411.10937, 2024

    2024 arXiv

  16. [24]

    EndoChat: Grounded multimodal large language model for endoscopic surgery,

    G. Wanget al., “EndoChat: Grounded multimodal large language model for endoscopic surgery,”Med. Image Anal., vol. 107, p. 103789, 2026. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S1361841525003354

    2026

  17. [25]

    Surgical-lvlm: Learning to adapt large vision-language model for grounded visual question answering in robotic surgery,

    G. Wanget al., “Surgical-lvlm: Learning to adapt large vision-language model for grounded visual question answering in robotic surgery,”arXiv preprint arXiv:2405.10948, 2024

    2024 arXiv

  18. [26]

    Surgvidlm: Towards multi-grained surgical video under- standing with large language model,

    G. Wanget al., “Surgvidlm: Towards multi-grained surgical video under- standing with large language model,”arXiv preprint arXiv:2506.17873, 2025

    2025

  19. [27]

    SurgVLM: A large vision-language model and sys- tematic evaluation benchmark for surgical intelligence,

    Z. Zenget al., “SurgVLM: A large vision-language model and sys- tematic evaluation benchmark for surgical intelligence,”arXiv preprint arXiv:2506.02555, 2025

    2025 arXiv

  20. [28]

    Surgpub-video: A comprehensive surgical video framework for enhanced surgical intelligence in vision-language model,

    Y . Liet al., “Surgpub-video: A comprehensive surgical video framework for enhanced surgical intelligence in vision-language model,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 8, pp. 6628–6635, Mar. 2026. [Online]. Available: https://doi.org/10....

    2026 doi

  21. [29]

    Sureon: A benchmark and vision-language-model for surgical reasoning,

    A. Perezet al., “Sureon: A benchmark and vision-language-model for surgical reasoning,”arXiv preprint arXiv:2603.06570, 2026

    2026

  22. [30]

    Surg-r1: A hierarchical reasoning foundation model for scalable and interpretable surgical decision support with multi-center clinical validation,

    J. Jianget al., “Surg-r1: A hierarchical reasoning foundation model for scalable and interpretable surgical decision support with multi-center clinical validation,”arXiv preprint arXiv:2603.12430, 2026

    2026

  23. [31]

    SurgLaVi: Large-scale hierarchical dataset for surgical vision- language representation learning,

    A. Perez, C. Nwoye, R. Raji Kermani, O. Mohareri, and M. Abdullah Jamal, “SurgLaVi: Large-scale hierarchical dataset for surgical vision- language representation learning,”Med. Image Anal., vol. 110, p. 103982, 2026. [Online]. Available: https://www.sciencedirect.com/ science/...

    2026

  24. [32]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,”arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  25. [33]

    Qwen3-vl technical report,

    S. Baiet al., “Qwen3-vl technical report,”arXiv preprint arXiv:2511.21631, 2025

    2025 arXiv

  26. [34]

    Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy,

    Z. Wanget al., “Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy,” in Proc. MICCAI. Springer, 2022, pp. 486–496

    2022

  27. [35]

    Challenges in multi-centric generalization: phase and step recognition in roux-en-y gastric bypass surgery,

    J. L. Lavanchyet al., “Challenges in multi-centric generalization: phase and step recognition in roux-en-y gastric bypass surgery,”Int. J. Comput. Assist. Radiol. Surg., vol. 19, no. 11, pp. 2249–2257, 2024

    2024

  28. [36]

    Comparative validation of machine learning al- gorithms for surgical workflow and skill analysis with the heichole benchmark,

    M. Wagneret al., “Comparative validation of machine learning al- gorithms for surgical workflow and skill analysis with the heichole benchmark,”Med. Image Anal., vol. 86, p. 102770, 2023

    2023

  29. [37]

    Pixel-wise recognition for holistic surgical scene understanding,

    N. Ayobiet al., “Pixel-wise recognition for holistic surgical scene understanding,”Med. Image Anal., vol. 106, p. 103726, 2025. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S1361841525002737

    2025

  30. [38]

    Sar-rarp50: Segmentation of surgical instrumen- tation and action recognition on robot-assisted radical prostatectomy challenge,

    D. Psychogyioset al., “Sar-rarp50: Segmentation of surgical instrumen- tation and action recognition on robot-assisted radical prostatectomy challenge,”arXiv preprint arXiv:2401.00496, 2023

    2023 arXiv

  31. [39]

    Rendezvous: Attention mechanisms for the recogni- tion of surgical action triplets in endoscopic videos,

    C. I. Nwoyeet al., “Rendezvous: Attention mechanisms for the recogni- tion of surgical action triplets in endoscopic videos,”Med. Image Anal., vol. 78, p. 102433, 2022

    2022

  32. [40]

    CIDEr: Consensus-based image description evaluation,

    R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-based image description evaluation,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2015, pp. 4566–4575

    2015

  33. [41]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” inProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP- IJCNLP), 201...

    2019

  34. [42]

    LoRA: Low-rank adaptation of large language models,

    E. J. Huet al., “LoRA: Low-rank adaptation of large language models,” inProc. ICLR, 2022. [Online]. Available: https://openreview.net/forum? id=nZeVKeeFYf9

    2022

  35. [43]

    Learning transferable visual models from natural language supervision,

    A. Radfordet al., “Learning transferable visual models from natural language supervision,” inProc. Int. Conf. Mach. Learn. (ICML). PmLR, 2021, pp. 8748–8763

    2021

  36. [44]

    Medgemma technical report,

    A. Sellergrenet al., “Medgemma technical report,”arXiv preprint arXiv:2507.05201, 2025

    2025 arXiv

  37. [2017]

    Available: http://dx.doi.org/10.1038/s41551-017-0132-7

    [Online]. Available: http://dx.doi.org/10.1038/s41551-017-0132-7

    doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.