Pith. sign in

REVIEW 4 major objections 6 minor 39 references

SurgXBench: Explainable Vision-Language Model Benchmark for Surgery

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Surgical vision-language models, even after surgical-video pretraining, often classify instruments correctly by attending to weak contextual cues rather than to the clinically relevant region.

desk verdict A useful explainability benchmark for surgical VLMs, but its headline 'weak cues' claim is confounded by different XAI methods across model groups. read the letter →

arxiv 2505.10764 v4 pith:BBEJIXN4 submitted 2025-05-16 cs.CV

Jiajun Cheng , Xianwu Zhao , Sainan Liu , Xiaofan Yu , Ravi Prakash , Patrick J. Codd , Jonathan Elliott Katz , Shan Lin This is my paper · ORCID
classification cs.CV
keywords surgicalvision-languagemodelsexplainableAIinstrumentclassificationtripletrecognitionattentionalignmentzero-shotevaluationlaparoscopicvideoGrad-CAM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that current vision-language models for surgery can produce correct instrument and action classifications while attending to the wrong parts of the image. It builds a benchmark that adds explainability analysis to standard accuracy metrics, scoring how much a model's attention overlaps with clinically relevant regions such as instrument boxes and instrument-tissue interaction areas. Across six general and surgical VLMs on two laparoscopic datasets, the paper finds that surgical models can beat general models on classification accuracy yet show worse attention alignment, meaning their correct answers are often not grounded in the visual evidence a clinician would use. If the claim holds, standard benchmarks overstate the reliability of surgical VLMs, and future training needs frame-level visual and reasoning supervision, not just video-level language alignment.

What carries the argument

The load-bearing mechanism is post-hoc explanation heatmaps paired with region-overlap scores. Grad-CAM produces heatmaps for ResNet-based contrastive VLMs; gradient-weighted attention rollout does the same for CLIP's vision transformer; CLEANN builds causal graphs from token-level conditional-independence tests for the large vision-language model. The alignment metrics AA and AC threshold the heatmap at the top 20% of attention and measure overlap with ground-truth instrument boxes, either for the predicted class or for any annotated object. For triplet recognition, RAFT optical flow plus a pan/tilt/zoom/roll camera-motion correction generates binary masks of instrument-tissue interaction, and the RAS scores measure attention overlap with those masks plus instrument regions. These scores are what let the study distinguish correct-by-luck from correct-for-the-right-reason predictions.

What would settle it

Run a controlled occlusion test on a held-out surgical video set: for each correct prediction, occlude the top 20% attention region and record how often the prediction survives. If correct predictions survive high-attention occlusion about as often as they survive occlusion of random patches, attention does not track decision evidence; if occlusion of top-attention regions reliably collapses accuracy while random occlusion does not, attention is diagnostic and the paper's central claim is false.

Watch

Extended reading notes

Core claim

The central claim is that for surgical instrument and triplet recognition, a model can be right by coincidence: correct classification correlates poorly with attention on the clinically informative regions. On Cholec80BBox, contrastive general VLMs such as BLIP achieve higher alignment scores (bag TP_AC 66.9%, grasper 32.1%) than surgical VLMs, while surgical models such as PeskaVLP lead in F1 for several instruments (hook 0.7410, bipolar 0.7899, irrigator 0.7962) but have low attention overlap (hook TP_AA 6.6%, TP_AC 7.6%). Occluding the regions the surgical model attends to does not reliably remove the correct triplet from the top-5 predictions: 28.2% of previously IVT-matched frames lost their match, and the average similarity score dropped by 9.87. Causal-graph analysis of a large vision-language model shows text tokens such as surgical, image, and instrument contributing more directly than visual tokens, with visual tokens scattered outside clinically relevant regions. The paper interprets these patterns as evidence that surgical VLMs lean on weak contextual cues and that both visual grounding and explicit reasoning supervision are needed.

Load-bearing premise

The conclusion collapses if post-hoc explanation heatmaps (Grad-CAM, attention rollout, CLEANN) do not faithfully reveal the visual evidence the model actually used; the paper itself notes that for large vision-language models Grad-CAM is only a soft interpretability measure.

Editorial extensions

If this is right

  • Standard classification metrics alone cannot certify a surgical VLM: a model can score well on F1 while its attention misses the instrument, so reliability reporting should include attention-alignment scores.
  • Surgical VLMs trained on clip-level video-language supervision need frame-level visual grounding, because weak supervision is the likely cause of reliance on contextual cues.
  • LVLMs used for surgery should be trained or prompted to reason from visual tokens rather than dominant text priors, since causal-graph evidence shows text tokens leading the prediction.
  • The explainability pipeline extends to other surgical tasks whenever clinically relevant cues can be defined, so phase recognition and tissue identification could use the same alignment scoring.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If heatmap faithfulness holds, a cheap safety check for surgical VLMs is to require a minimum alignment score before a prediction is shown to a surgeon; accuracy alone should not gate alerts.
  • The paper's logic suggests a testable division of labor: pair a general VLM that localizes well with a surgical VLM that classifies well, potentially outperforming either alone.
  • The occlusion protocol could itself become a benchmark measure: report the fraction of correct predictions that survive occlusion of their high-attention regions, since a high survival rate means the explained evidence is not what drives the model.
  • The RAFT-based motion-mask idea transfers to non-surgical video tasks where action location matters, such as assembly or manipulation, using the same camera-motion correction to isolate true interaction regions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. SurgXBench introduces an explainability-oriented benchmark for zero-shot evaluation of six vision-language models (three surgical, three general) on surgical instrument classification (Cholec80BBox) and triplet recognition (CholecT45). The paper proposes attention-alignment metrics (AA, AC, RAS), uses RAFT with camera-motion correction to localize instrument-tissue interactions, applies Grad-CAM, gradient-weighted attention rollout, and CLEANN for visualization, and reports attention-based occlusion and causal-graph analyses. The central claim is that surgical VLMs, despite domain-specific training, often rely on weak contextual cues rather than clinically relevant visual evidence, and that general contrastive VLMs can localize instruments better even when their classifications are incorrect.

Significance. If the central finding is valid, the paper addresses an important gap: standard accuracy/F1 metrics can overstate the reliability of surgical VLMs, and the proposed benchmark would provide a useful template for trustworthiness evaluation in medical vision-language systems. The paper makes concrete contributions: a public benchmark with explainability metrics, automatic extraction of interaction regions via RAFT with camera-motion correction, qualitative causal graphs, and an occlusion-based counterfactual analysis. The authors are appropriately careful in some places, notably in acknowledging the softness of Grad-CAM interpretations for LVLMs in Supplementary A.3.1. However, the headline cross-model conclusion depends on contested post-hoc heatmap faithfulness and on a comparison that confounds explanation method with model family; the currently reported evidence is not yet strong enough to support the broad claim as stated.

major comments (4)
  1. [§3.2, §3.4.1, Fig. 3] The cross-model AA/AC comparison is confounded by explanation method. Surgical VLMs (ResNet backbones) are explained with Grad-CAM (Eq. 6), while CLIP and BLIP (ViT backbones) use gradient-weighted attention rollout (Eq. 7), and LLaVA uses a different Grad-CAM variant plus CLEANN. These methods have different spatial resolutions and biases, so lower AA/AC for the surgical group could reflect the explanation method rather than weaker use of clinically relevant evidence. The paper should provide a method-controlled comparison, for example by applying the same explanation method to all models where feasible, or by reporting results separately for each explanation method and showing that conclusions are invariant.
  2. [§5.1, §5.2, Fig. 3, Fig. 4] The alignment scores are reported as point estimates or box plots without confidence intervals or statistical tests. Claims such as "general contrastive learning VLMs outperform surgical VLMs in attention alignment" rest on absolute differences that may be within sampling variability, especially given the small number of videos (5 for Cholec80BBox; 2 for the RAS analysis). The authors should report error bars, per-frame variability, and significance tests, and should include a sensitivity analysis over the hand-chosen thresholds τ, γ, δ, and top-k (Supp C.1) to show that the main conclusions do not depend on these choices.
  3. [§5.4, Fig. 6, Supp E.2] The occlusion analysis lacks a random-occlusion control. Reporting that occluding high-attention regions removes IVT matches is only informative if compared with occluding randomly selected regions at the same area fraction; without such a control, the observed drop could be a generic effect of image occlusion. In addition, the analysis is applied to surgical VLMs but not to the general VLMs, so it cannot support the cross-model comparison that is central to the paper's conclusion. The authors should add a random-occlusion baseline and, if possible, apply the same counterfactual procedure to all model families.
  4. [Supp A.3.1, §5.3, Fig. 5] The paper explicitly concedes that Grad-CAM for LVLMs "does not fully capture the underlying reasoning" and serves only as a "soft interpretability measure." Given that the central claim includes LLaVA and LVLMs, the causal graph analysis (two illustrative examples in Fig. 5) is too thin to carry quantitative weight. The authors should either derive quantitative metrics from CLEANN across a larger set of examples, or explicitly restrict the central claim to contrastive models and present the LVLM results as preliminary qualitative observations.
minor comments (6)
  1. [Introduction] There is a typo: "robustnees" should be "robustness."
  2. [Related Work] The phrase "the only existing VLM benchmark in surgery is by by Rauet al." contains a duplicated "by".
  3. [Eq. (2)-(3) context] In the definition of Nk, "appearnces" should be "appearances."
  4. [§4.3.2] The heading "Triplet Recongition Task" contains a misspelling; it should be "Triplet Recognition Task."
  5. [Supp E.1] The phrase "Addition results include pericison, recall and box plot" should be "Additional results include precision, recall and box plots."
  6. [§3.2, Fig. 1] The paper uses "attention" to refer to heatmaps from Grad-CAM, which in ResNet backbones are gradient-based saliency maps rather than native attention; using a distinct term such as "saliency" or "attribution" would avoid confusion with true attention mechanisms.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SurgXBench's central claim is an empirical finding from fixed pretrained VLMs, not a construction from its own metrics.

full rationale

The paper's derivation chain is self-contained in the relevant sense: the six benchmarked VLMs are frozen pretrained systems, evaluated zero-shot against external annotations (Cholec80BBox, CholecT45), and no parameter is fitted to produce the conclusion that surgical VLMs rely on weak contextual cues. The proposed AA, AC, and RAS metrics are thresholded overlap measures between post-hoc attention heatmaps and annotated regions or RAFT-derived motion masks; they are reported as measurements of the fixed model outputs, and the thresholds (tau=0.8, gamma=0.8, delta=0.9, top-k=5) are hand-chosen constants, not fitted to force the weak-cue result. The central claim that surgical VLMs show worse attention alignment is an empirical observation: for example, PeskaVLP achieves higher F1 for multiple classes yet lower TP_AA/TP_AC scores, and the occlusion study shows predictions surviving occlusion of relevant regions. These could be wrong if Grad-CAM or attention rollout is unfaithful, and the paper itself concedes in Supplementary A.3.1 that Grad-CAM for LVLMs 'does not fully capture the underlying reasoning' and 'serves more as a soft interpretability measure'; that is a correctness and validity threat, not circularity, because the metric values are not logically entailed by the method definitions. The only self-citations ([15], [16]) appear in a list of prior instrument-segmentation work in the introduction and are not load-bearing for any uniqueness, ansatz, or prediction step. No equation equates a predicted quantity to an input annotation, and no cited prior result by the same authors is invoked to forbid alternatives. Thus there is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on two pillars: (1) post-hoc XAI heatmaps indicating model attention, and (2) RAFT-derived motion masks indicating instrument-tissue interactions. Both are strong modeling assumptions, and neither is validated against ground-truth reasoning. The hand-chosen thresholds (tau, gamma, delta, top-k) influence all reported alignment scores. No new physical or conceptual entities are introduced.

free parameters (5)
  • tau (attention threshold) = 0.8
    Top 20% of attention values define the predicted attention region in AA, AC, and RAS; hand-chosen, affects all alignment scores.
  • gamma (RAFT motion threshold) = 0.8
    Percentile threshold on corrected RAFT flow used to build interaction masks; hand-chosen, directly affects RAS scores.
  • delta (similarity percentile) = 0.9
    Classes with similarity score above the 90th percentile are predicted present; hand-chosen, changes precision/recall and the set of TP/FP predictions used for AA/AC.
  • top-k for triplet predictions = 5
    Only the top 5 prompt scores are considered as triplet predictions; hand-chosen, changes IVT/IV/IT match rates.
  • CLEANN parameters (kappa, alpha, n, top-5 heads) = 0.01, 1e-5, 128, 5
    Hand-chosen thresholds for the causal graph analysis; these affect the qualitative causal graphs and any conclusions about text-token reliance.
assumptions (4)
  • domain assumption Post-hoc XAI heatmaps (Grad-CAM, attention rollout, CLEANN) faithfully indicate which image regions the model used for its prediction.
    All AA, AC, and RAS scores are computed from these heatmaps (Sec 3.2, 3.4). The paper concedes in Supp A.3.1 that for LVLMs the analysis 'serves more as a soft interpretability measure.'
  • domain assumption RAFT optical flow with linear pan/tilt/zoom/roll camera motion correction identifies instrument-tissue interaction regions.
    RAS uses RAFT-derived masks as supervision for where actions occur (Sec 3.4.2, Supp B). If the camera model is misspecified or high motion does not equal interaction, the scores are biased.
  • domain assumption Ground-truth instrument bounding boxes and RAFT motion regions are an adequate definition of clinically relevant cues.
    AA and AC define relevance as overlap with instrument boxes; RAS uses boxes plus motion masks. Relevant tissue deformation or interaction context outside these regions is excluded by construction.
  • domain assumption Thresholded similarity scores define VLM predictions in zero-shot evaluation.
    The paper predicts a class when its similarity score is above the 90th percentile (delta=0.9) and takes top-5 for triplets (Sec 4.3, Supp C). This assumes score magnitude and rank are meaningful decision indicators.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SurgXBench: Explainable Vision-Language Model Benchmark for Surgery." pith.science (2026). https://pith.science/paper/BBEJIXN4

@misc{pith2026250510764,
  author       = {Pith},
  title        = {Pith review of: SurgXBench: Explainable Vision-Language Model Benchmark for Surgery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BBEJIXN4}},
  note         = {Machine review of arXiv:2505.10764}
}
read the original abstract

Innovations in digital intelligence are transforming robotic surgery with more informed decision-making. Real-time awareness of surgical instrument presence and actions (e.g., cutting tissue) is essential for such systems. Yet, despite decades of research, most machine learning models for this task are trained on small datasets and still struggle to generalize. Recently, vision-Language Models (VLMs) have brought transformative advances in reasoning across visual and textual modalities. Their unprecedented generalization capabilities suggest great potential for advancing intelligent robotic surgery. However, surgical VLMs remain under-explored, and existing models show limited performance, highlighting the need for benchmark studies to assess their capabilities and limitations and to inform future development. To this end, we benchmark the zero-shot performance of several advanced VLMs on two public robotic-assisted laparoscopic datasets for instrument and action classification. Beyond standard evaluation, we integrate explainable AI to visualize VLM attention and uncover causal explanations behind their predictions. This provides a previously underexplored perspective in this field for evaluating the reliability of model predictions. We also propose several explainability analysis-based metrics to complement standard evaluations. Our analysis reveals that surgical VLMs, despite domain-specific training, often rely on weak contextual cues rather than clinically relevant visual evidence, highlighting the need for stronger visual and reasoning supervision in surgical applications.

Figures

Figures reproduced from arXiv: 2505.10764 by the authors.

Figure 1
Figure 1. We evaluate two general VLMs, CLIP-ViT-B/32 [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pipeline of extracting instrument-tissue interac [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Instrument classification results of General VLMs (CLIP[ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Triplet recognition results on CholecT45[ [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: LLaVA predictions with CLEANN-generated causal graphs (radius 3, top-5 attention heads). (a): Correct [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Counterfactual occlusion analysis results on SurgVLP. We picked two frames with the two most common [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Surgical instrument detection performance and attention alignment across seven instrument types from the [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Occlusion analysis results for SurgVLP, HecVLP, and PeskaVLP. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: This figure shows triplet recognition examples from LLaVA [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 30 canonical work pages

  1. [1]

    Almeida, R

    J. Almeida, R. Minetto, T. A. Almeida, R. da S. Torres, and N. J. Leite. Robust estimation of camera motion using optical flow models. InProceedings of the International Symposium on Visual Computing, pages 435–446, 2009. 8

    work page 2009

  2. [2]

    Abdulbaki Alshirbaji, H

    T. Abdulbaki Alshirbaji, H. Arabian, N. A. Jalal, A. Battistel, P. D. Docherty, T. Neumuth, and K. Moeller. Cholec80-boxes: Bounding-box labeling data for surgical tools in cholecystectomy images

    work page

  3. [3]

    Vision-based and marker-less surgical tool detection and tracking: a review of the literature

    David Bouget, Max Allan, Danail Stoyanov, and Pierre Jannin. Vision-based and marker-less surgical tool detection and tracking: a review of the literature. Medical image analysis, 35:633–654, 2017

    work page 2017

  4. [4]

    Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks

    Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 839–847, 2018

    work page 2018

  5. [5]

    Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers

    Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. InPro- ceedings of the IEEE/CVF international conference on computer vision, pages 397–406, 2021

    work page 2021

  6. [6]

    J. Choe, S. J. Oh, S. Lee, S. Chun, Z. Akata, and H. Shim. Evaluating weakly supervised object localization methods right. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3133–3142, 2020

    work page 2020

  7. [7]

    Robotic surgery.Nature Reviews Bioengineering, pages 1–14, 2025

    Gastone Ciuti, Robert J Webster III, Ka-Wai Kwok, and Arianna Menciassi. Robotic surgery.Nature Reviews Bioengineering, pages 1–14, 2025

    work page 2025

  8. [8]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis- senborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. InInterna- tional Conference on Learning Representations, 2021

    work page 2021

Show all 39 references
  1. [9]

    A decade retrospective of medical robotics research from 2010 to 2020.Science robotics, 6(60):eabi8017, 2021

    Pierre E Dupont, Bradley J Nelson, Michael Goldfarb, Blake Hannaford, Arianna Menciassi, Marcia K O’Malley, Nabil Simaan, Pietro Valdastri, and Guang-Zhong Yang. A decade retrospective of medical robotics research from 2010 to 2020.Science robotics, 6(60):eabi8017, 2021

    2010

  2. [10]

    Toolnet: holistically-nested real-time segmentation of robotic surgical tools

    Luis C Garcia-Peraza-Herrera, Wenqi Li, Lucas Fidon, Caspar Gruijthuijsen, Alain Devreker, George Attilakos, Jan Deprest, Emmanuel Vander Poorten, Danail Stoyanov, Tom Vercauteren, et al. Toolnet: holistically-nested real-time segmentation of robotic surgical tools. In2017 IEE...

    2017

  3. [11]

    Goyal, Z

    Y . Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, and S. Lee. Counterfactual visual explanations. In Proceedings of the International Conference on Machine Learning (ICML), pages 2376–2384, 2019

    2019

  4. [12]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016

    2016

  5. [13]

    N. A. Jalal, T. A. Alshirbaji, P. D. Docherty, H. Arabian, T. Neumuth, and K. Möller. Surgical tool classification & localisation using attention and multi-feature fusion deep learning approach. IFAC-PapersOnLine, 56(2):5626–5631, 2023

    2023

  6. [14]

    Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. InProceedings of the International Conference on Machine Learning (ICML), pages 12888–12900, 2022

    2022

  7. [15]

    Lc-gan: Image- to-image translation based on generative adversarial network for endoscopic images

    Shan Lin, Fangbo Qin, Yangming Li, Randall A Bly, Kris S Moe, and Blake Hannaford. Lc-gan: Image- to-image translation based on generative adversarial network for endoscopic images. In2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2914–2...

    2020

  8. [16]

    Multi-frame feature aggregation for real-time instrument seg- mentation in endoscopic video.IEEE Robotics and Automation Letters, 6(4):6773–6780, 2021

    Shan Lin, Fangbo Qin, Haonan Peng, Randall A Bly, Kris S Moe, and Blake Hannaford. Multi-frame feature aggregation for real-time instrument seg- mentation in endoscopic video.IEEE Robotics and Automation Letters, 6(4):6773–6780, 2021

    2021

  9. [17]

    Visual instruction tuning.Ad- vances in Neural Information Processing Systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Ad- vances in Neural Information Processing Systems, 36:34892–34916, 2023

    2023

  10. [18]

    Attention-guided lightweight network for real-time segmentation of robotic surgical instruments

    Zhen-Liang Ni, Gui-Bin Bian, Zeng-Guang Hou, Xiao-Hu Zhou, Xiao-Liang Xie, and Zhen Li. Attention-guided lightweight network for real-time segmentation of robotic surgical instruments. In 2020 IEEE international conference on robotics and automation (ICRA), pages 9939–9945. IEEE, 2020

    2020

  11. [19]

    C. I. Nwoye, T. Yu, C. Gonzalez, B. Seeliger, P. Mascagni, D. Mutter, J. Marescaux, and N. Padoy. Rendezvous: Attention mechanisms for the recogni- tion of surgical action triplets in endoscopic videos. Medical Image Analysis, 78:102433, 2022

    2022

  12. [20]

    Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation.arXiv preprint arXiv:2010.16061, 2020

    David MW Powers. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation.arXiv preprint arXiv:2010.16061, 2020. 9

    2010 arXiv

  13. [21]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference ...

    2021

  14. [22]

    Systematic evaluation of large vision-language models for surgical artificial intelligence.arXiv preprint arXiv:2504.02799, 2025

    Anita Rau, Mark Endo, Josiah Aklilu, Jaewoo Heo, Khaled Saab, Alberto Paderno, Jeffrey Jopling, F Christopher Holsinger, and Serena Yeung-Levy. Systematic evaluation of large vision-language models for surgical artificial intelligence.arXiv preprint arXiv:2504.02799, 2025

    2025 arXiv

  15. [23]

    R. Y . Rohekar, Y . Gurwicz, and S. Nisimov. Causal in- terpretation of self-attention in pre-trained transform- ers. InAdvances in Neural Information Processing Systems, volume 36, pages 31450–31465, 2023

    2023

  16. [24]

    Comparative validation of multi-instance instrument segmentation in endoscopy: results of the robust-mis 2019 challenge.Medical image analysis, 70:101920, 2021

    Tobias Roß, Annika Reinke, Peter M Full, Martin Wagner, Hannes Kenngott, Martin Apitz, Hellena Hempe, Diana Mindroc-Filimon, Patrick Scholz, Thuy Nuong Tran, et al. Comparative validation of multi-instance instrument segmentation in endoscopy: results of the robust-mis 2019 ch...

    2019

  17. [25]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. InProceedings of the IEEE International Conference on Computer Vision (ICCV), pages 61...

    2017

  18. [26]

    G. B. M. Stan, E. Aflalo, R. Y . Rohekar, et al. Lvlm-interpret: An interpretability tool for large vision-language models.arXiv preprint arXiv:2404.03118, 2024

    2024 arXiv

  19. [27]

    Teed and J

    Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419, 2020

    2020

  20. [28]

    Endonet: A deep architecture for recognition tasks on laparoscopic videos.IEEE Transactions on Medical Imaging, 36(1):86–97, 2016

    Apolline Sauzéon Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel de Mathelin, and Nicolas Padoy. Endonet: A deep architecture for recognition tasks on laparoscopic videos.IEEE Transactions on Medical Imaging, 36(1):86–97, 2016

    2016

  21. [29]

    Cholect50: A benchmark for surgical action triplet recognition.arXiv preprint arXiv:2204.05235, 2022

    Arnaud Twinanda, Emre Aksan, Raphael Sznitman, Peter Wolf, Leo Gruendlinger, Christoph Jud, and Tiago da Costa. Cholect50: A benchmark for surgical action triplet recognition.arXiv preprint arXiv:2204.05235, 2022

    2022 arXiv

  22. [30]

    Yuille, and Wei Shen

    Haofan Wang, Zifan Wang, Pengfei Zhao, Yanfeng Pei, Xiaoyu Wang, Alan L. Yuille, and Wei Shen. Score-cam: Score-weighted visual explanations for convolutional neural networks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages ...

    2020

  23. [31]

    The robot will see you now: Foundation models are the path forward for autonomous robotic surgery.Science Robotics, 10(104):eadt0684, 2025

    Michael Yip. The robot will see you now: Foundation models are the path forward for autonomous robotic surgery.Science Robotics, 10(104):eadt0684, 2025

    2025

  24. [32]

    Hecvl: Hierarchical video-language pretraining for zero-shot surgical phase recognition

    Kun Yuan, Vinkle Srivastav, Nassir Navab, and Nicolas Padoy. Hecvl: Hierarchical video-language pretraining for zero-shot surgical phase recognition. arXiv preprint arXiv:2405.10075, 2024

    2024 arXiv

  25. [33]

    Procedure-aware surgical video-language pretraining with hierarchical knowledge augmenta- tion

    Kun Yuan, Vinkle Srivastav, Nassir Navab, and Nico- las Padoy. Procedure-aware surgical video-language pretraining with hierarchical knowledge augmenta- tion. InProceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  26. [34]

    La- vanchy, Jacques Marescaux, Pietro Mascagni, Nassir Navab, and Nicolas Padoy

    Kun Yuan, Vinkle Srivastav, Tong Yu, Joël L. La- vanchy, Jacques Marescaux, Pietro Mascagni, Nassir Navab, and Nicolas Padoy. Learning multi-modal rep- resentations by watching hundreds of surgical video lectures.arXiv preprint arXiv:2307.15220, 2023

    2023 arXiv

  27. [35]

    Zablocki, H

    É. Zablocki, H. Ben-Younes, P. Pérez, and M. Cord. Explainability of deep vision-based autonomous driv- ing systems: Review and challenges.International Journal of Computer Vision, 130(10):2425–2452, 2022

    2022

  28. [36]

    Zeiler and Rob Fergus

    Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. InProceed- ings of the European Conference on Computer Vision (ECCV), pages 818–833. Springer, 2014

    2014

  29. [37]

    Vision-language models for vision tasks: A survey.IEEE transactions on pattern analysis and machine intelligence, 46(8):5625–5644, 2024

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey.IEEE transactions on pattern analysis and machine intelligence, 46(8):5625–5644, 2024

    2024

  30. [38]

    Zhang, Y

    X. Zhang, Y . Quan, C. Shen, X. Yuan, S. Yan, L. Xie, W. Wang, C. Gu, H. Tang, and J. Ye. From redundancy to relevance: Information flow in lvlms across reason- ing tasks.arXiv preprint arXiv:2406.06579, 2024

    2024 arXiv

  31. [39]

    What surgical tools do you see? Choose from: Grasper, Bipolar, Hook, Scissors, Clipper, Irrigator, Bag

    Yiming Zhang, Ying Weng, and Jonathan Lund. Applications of explainable artificial intelligence in diagnosis and surgery.Diagnostics, 12(2):237, 2022. 10 Supplementary Material A Post-hoc XAI Visualization Tech- niques - Mathematical Formula- tions This section provides the de...

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.