Pith. sign in

REVIEW 3 major objections 3 minor 70 references

Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AI

T0 review · 3 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Surgical videos become structured reports through a three-stage transformer pipeline, with 96% tool-detection precision and BERT score 0.74.

desk verdict The evaluation protocol sinks an otherwise competent pipeline: frame-level split leaks near-duplicate frames across train/test, and the caption references are templates built from the model's own input labels. read the letter →

arxiv 2504.19918 v1 pith:X6DBCJGP submitted 2025-04-28 cs.CV cs.AI

Hugo Georgenthum , Cristian Cosentino , Fabrizio Marozzo , Pietro Liò This is my paper · ORCID
classification cs.CVcs.AI
keywords surgicalvideosummarizationmultimodaltransformersVisionTransformerlaparoscopiccholecystectomyCholecT50largelanguagemodelsreportgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automatic summarization of surgical videos matters for training, documentation, and post-operative review, and this paper proposes a three-stage pipeline to produce such reports from raw laparoscopic footage. The pipeline first labels instruments and anatomical targets in each frame with a Vision Transformer, then converts those labels into frame-level captions by fusing visual and textual embeddings, then uses a ViViT-based temporal encoder to generate clip-level summaries, and finally feeds those summaries to GPT-4 to compose a structured operative report. The authors report 96% precision in tool detection and BERT scores of about 0.74–0.81 for the captioning stages, arguing that the modular design keeps each decision step inspectable and reduces hallucination. A sympathetic reading is that a hierarchy of vision, temporal modeling, and language generation is enough to translate continuous surgical video into coherent clinical text.

What carries the argument

The load-bearing mechanism is the hierarchical fusion of visual and textual features across three time scales, carried by two named architectures: ViT for single-frame patches and ViViT, a video vision transformer that extends spatial patches with a temporal dimension, for clips of 32 frames with 16-frame overlap. Frame captions are produced by a cross-attention decoder (T5) over concatenated ViT and DistilBERT embeddings; clip captions use ViViT's spatiotemporal tokens fused with the frame captions; and the final report is generated by GPT-4 from a prompt that instructs it to merge repeated activities and combine their durations. The second part of the mechanism is two-phase training, in which the frame captioner is fine-tuned on the object detector's predicted labels and the clip captioner on the frame captioner's outputs, so each stage learns to tolerate the noise of the stage before it.

What would settle it

Take any held-out surgical video, run the pipeline, and compare tool detection precision on clips that share no frames with training, and the BERT score of clip captions against independently written surgeon reports instead of the templated captions; a substantial drop in either number would show the reported performance depends on frame leakage and template-matching rather than genuine generalization.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that surgical video can be turned into a structured written report by a cascaded multimodal transformer, where each stage consumes the previous stage's output rather than raw video alone. Frame-level object detection identifies tools and targets; frame captions are generated by concatenating ViT visual embeddings with DistilBERT object-label embeddings and decoding with T5; clip captions come from a ViViT model that sees 32-frame clips plus the generated frame captions; and GPT-4 merges the clip captions into a single narrative with phase durations. The reported evidence is that this cascade reaches 96% precision in instrument detection, BLEU 0.73 for frame captions, BLEU 0.67 for clip captions, and improves further after fine-tuning the captioners on detector outputs instead of ground-truth labels, which the paper presents as a way to absorb error propagation.

Load-bearing premise

The reported generalization assumes a random frame-level split treats frames from the same surgical video as independent samples, and that templated captions derived from the annotations are valid ground truth for judging summary quality; if either fails, the headline numbers overstate real-world performance.

Editorial extensions

If this is right

  • If the reported precision and BERT scores hold, per-frame tool and target labeling at one frame per second is accurate enough to feed a captioning chain that produces readable phase-by-phase operative notes.
  • Fine-tuning each captioning stage on the previous stage's predicted outputs rather than gold labels closes most of the quality gap caused by error propagation.
  • The same three-stage object-to-frame-to-clip-to-report architecture should transfer to other surgical procedures once phase, action, and instrument annotations exist for training.
  • Because the final report is built from intermediate captions, it becomes possible to audit the report by inspecting the tool detections and clip captions, supporting explainability in clinical use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The evaluation splits frames, not videos, so adjacent frames from the same procedure appear in both training and test sets; the reported 96% tool precision therefore likely measures near-duplicate frame recognition rather than generalization to a new patient's surgery, and a video-level split would be the decisive test.
  • The reference frame and clip captions are templated sentences built from the same action, target, and phase annotations the model receives as input, so BLEU and BERTScore here measure how faithfully the model reproduces the template, not whether the summary is clinically true or complete.
  • Because the final report is written by GPT-4 from captions, a surgeon would be verifying the narrative against the captions, not the raw video; the trust boundary of the system is therefore at the clip-caption stage.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The manuscript proposes a three-stage pipeline for automatic surgical documentation from laparoscopic videos: (1) multi-label object classification of instruments and targets from individual video frames using a Vision Transformer (ViT); (2) frame-level caption generation with a ViT/DistilBERT/T5 cross-attention model, followed by clip-level caption generation with a ViViT/FLAN-T5 model that fuses visual and textual features; and (3) synthesis of a full surgical report by prompting GPT-4 with the concatenated clip captions. The system is evaluated on CholecT50, and the authors report strong instrument/target recognition, frame- and clip-level captioning scores (BLEU, ROUGE, BERTScore), and an example GPT-4 generated report. The central claim is that the modular, temporally aware architecture is capable of accurate tool detection and coherent, clinically usable surgical report generation.

Significance. If the reported results were valid, the paper would make a useful engineering contribution to automated surgical documentation: it integrates established components (ViT, ViViT, T5, GPT-4) in a modular pipeline, publicly releases preprocessing scripts and trained models, and describes an error-aware training strategy that is a reasonable practical idea. However, the evaluation protocol has two fundamental flaws that invalidate the main empirical claims: a frame-level train/test split that leaks near-duplicate frames from the same surgical videos into both training and test sets, and captioning references that are deterministic templates generated from the very annotations the models receive as input. Because of these flaws, the reported detection, captioning, and summarization numbers cannot be interpreted as evidence of generalization or of semantic quality.

major comments (3)
  1. [Section 4.1] The train/test/validation split is performed at the frame level (80/10/10) rather than at the video level. CholecT50 consists of only 50 videos, the frames are extracted at 1 fps, and clips are created with 32 frames and a 16-frame overlap. Consequently, frames and overlapping clips from the same surgical video appear in both the training and test sets, and adjacent frames are near-duplicates. The model can memorize video-specific appearance, instrument lighting, and phase transitions instead of learning generalizable tool recognition or action captioning. This temporal leakage invalidates the object-detection mAP in Figure 7, the frame-caption metrics in Table 2, the clip-caption metrics in Table 3, and the qualitative GPT-4 report assessment in Section 4.6, all of which are central to the paper's claims.
  2. [Section 4.1] The frame and clip captions used as both training targets and evaluation references are 'created artificially using the annotations' (verbs, targets, phases), and the captioning models receive exactly those annotations as part of their input, as described in Sections 3.2 and 4.2.2. The reference captions are therefore deterministic templates built from the same information the model consumes. The BLEU, ROUGE, and BERTScore numbers thus measure how well the model reconstructs a fixed template, not how accurately or usefully it describes surgical content. An evaluation with human-authored references and a protocol that prevents the reference content from being fed as input is needed to support the captioning claims.
  3. [Abstract, Figure 7, Tables 2-3] The abstract reports '96% precision in tool detection' and 'BERT score of 0.74', but no table or figure in the body reports a precision of 0.96 or a BERTScore of 0.74. Figure 7 shows mean average precision values for instruments and targets, Table 2 lists BERT F1 up to 0.8052 for frame captions, and Table 3 reports BERT F1 0.7607 for the robust clip model. The headline numbers in the abstract are not traceable to any result shown in the experimental section, making the main advertised findings unverifiable from the manuscript.
minor comments (3)
  1. [Section 3.1] The method described in Equations (2)-(4) is multi-label image classification, not object detection in the standard sense, since no bounding boxes or localizations are produced. The repeated use of 'object detection' and the mAP comparison in Figure 7 should be clarified or renamed to avoid ambiguity.
  2. [Figure 7] The bar chart does not show numerical values on the bars, and the legend order is not aligned with the bar grouping, making it difficult to determine which bars correspond to which model and metric. Please add value labels and a clearer legend.
  3. [Section 4.1] The sentence 'With a total of89827, the the clips are created...' contains typographical errors and a wrongly placed number, and the text should be revised for readability. Appendix A also contains the misspelling 'gallbloddger' in a predicted caption, which should be corrected.

Circularity Check

2 steps flagged · score 6.0 of 10

Captioning evaluation is partly circular: references are annotation-derived templates, and clip-caption targets are concatenations of the very frame captions fed as input.

  1. self definitional [Section 4.1 (Dataset and Preprocessing) and Section 4.4 (Table 2)]
    "Frame caption and clip captions are not provided in the original dataset, and are created artificially using the annotations. To create the frame captions, the verb, target and phase are used to construct a simple sentence presenting what the surgeon is doing in the frame. The same goes for the clip caption construction; the frame captions are concatenated into one clip captions, taking into account the time and order each action is taken during the clip."

    The reference text used for BLEU/ROUGE/BERTScore in Tables 2 and 3 is a deterministic template built from the CholecT50 verb/target/phase annotations. The frame-caption model is trained and evaluated on this annotation-derived text while receiving the same annotation lineage through its object-label encoder and video features; the clip-caption model receives frame captions as its textual modality. Thus the captioning metrics measure reconstruction of labels re-rendered as sentences, not agreement with independent human-written descriptions. The paper itself states the captions 'are created artificially using the annotations', so the evaluated target is defined from the model's input annotation stream rather than from external ground truth.

  2. self definitional [Section 4.1 and Section 4.5 (Table 3)]
    "The same goes for the clip caption construction; the frame captions are concatenated into one clip captions, taking into account the time and order each action is taken during the clip. ... The Model (GT) configuration takes ground-truth frame captions as input. This model shows substantial improvements in all evaluated metrics, confirming that high-quality intermediate textual features strongly enhance the performance of the clip-level caption generator."

    By the paper's own construction, the clip-caption reference is the concatenation of the frame captions with timing/order information. In the GT configuration the model is given those exact ground-truth frame captions as its textual input (Section 4.2.2 defines CC(Clip, F) with ground-truth F). The target is therefore a deterministic function of the input text, so high BLEU/ROUGE/BERTScore values show the model can copy or lightly rewrite its input rather than perform independent temporal summarization. The Generated configuration only replaces F with templated model outputs, so it stays inside the same annotation-to-template loop. No independent human-written clip reference is introduced.

full rationale

The object-detection stage is a standard supervised benchmark against external CholecT50 labels and is not itself circular; the ViT/ViViT/T5 machinery is publicly pretrained and the prediction equations (3)-(5) do not assume the target. The central captioning and summarization claims, however, are evaluated against references that the paper admits are 'created artificially using the annotations'. Frame captions are sentences built from verb/target/phase labels, and clip captions are built by concatenating frame captions; the same annotation stream is fed into the captioning models. Consequently the reported BLEU, ROUGE and BERTScore gains in Tables 2-3 reduce, in part, to reproducing templates derived from the input labels. This is a partial self-definitional circularity: the captioning 'predictions' are scored against their own construction inputs. The GPT-4 report stage is qualitatively evaluated and consumes those same templated clip captions, so it inherits the circular reference rather than adding independent evidence. The separate frame-level 80/10/10 split of CholecT50 (1 fps frames from only 50 videos, with overlapping 32-frame clips) is a serious external-validity problem, but it is a data-leakage/correctness issue rather than a circularity of the derivation chain; it is not counted in the circularity score. Self-citations [10,11] appear only to support the choice of GPT-4 and are not load-bearing for the method. Overall, the pipeline has real independent components, but the paper's headline captioning/summarization evidence is not self-contained because its evaluation references are generated from the same annotations that drive the model inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical or architectural entities are introduced. The load-bearing assumptions are the frame-level data split and the validity of templated captions as evaluation targets. The two fitted parameters (threshold, temperature) and the clip length hyperparameter are the main free choices, all tuned on validation data.

free parameters (3)
  • object detection threshold = 0.5 (initially), calibrated later
    Equation (4) uses a threshold to determine detected objects; Section 4.3 says the 0.5 threshold was selected based on empirical evaluation, so it is tuned on the validation set.
  • temperature scaling T = 1.8584
    Section 4.3 reports T = 1.8584 found by minimizing negative log-likelihood on a validation set (Eq. 16). This fitted parameter is used to report the improved calibration ECE.
  • clip length and overlap = 32 frames, 16 overlap
    Sections 3 and 4.1 choose 32-frame clips with 16-frame overlap. This hyperparameter affects all clip-level results and temporal modeling.
assumptions (3)
  • domain assumption CholecT50 annotations (instruments, verbs, targets, phases) are accurate and complete ground truth.
    The entire training and evaluation depend on these labels; no manual verification or noise analysis is reported. Section 4.1.
  • ad hoc to paper Frame-level random split is a valid generalization test for surgical videos.
    Section 4.1 splits by frames rather than videos, implicitly assuming frames from the same video are independent. This is false because adjacent frames are near-duplicates, yet all reported numbers rely on it.
  • ad hoc to paper Template sentences constructed from annotations are valid references for evaluating summarization quality.
    Section 4.1 creates frame and clip captions artificially from annotations and uses them as evaluation references. This assumes the templates capture true semantic quality, but they are deterministic string constructions from the same labels the model sees as input.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AI." pith.science (2026). https://pith.science/paper/X6DBCJGP

@misc{pith2026250419918,
  author       = {Pith},
  title        = {Pith review of: Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X6DBCJGP}},
  note         = {Machine review of arXiv:2504.19918}
}
read the original abstract

The automatic summarization of surgical videos is essential for enhancing procedural documentation, supporting surgical training, and facilitating post-operative analysis. This paper presents a novel method at the intersection of artificial intelligence and medicine, aiming to develop machine learning models with direct real-world applications in surgical contexts. We propose a multi-modal framework that leverages recent advancements in computer vision and large language models to generate comprehensive video summaries. % The approach is structured in three key stages. First, surgical videos are divided into clips, and visual features are extracted at the frame level using visual transformers. This step focuses on detecting tools, tissues, organs, and surgical actions. Second, the extracted features are transformed into frame-level captions via large language models. These are then combined with temporal features, captured using a ViViT-based encoder, to produce clip-level summaries that reflect the broader context of each video segment. Finally, the clip-level descriptions are aggregated into a full surgical report using a dedicated LLM tailored for the summarization task. % We evaluate our method on the CholecT50 dataset, using instrument and action annotations from 50 laparoscopic videos. The results show strong performance, achieving 96\% precision in tool detection and a BERT score of 0.74 for temporal context summarization. This work contributes to the advancement of AI-assisted tools for surgical reporting, offering a step toward more intelligent and reliable clinical documentation.

Figures

Figures reproduced from arXiv: 2504.19918 by the authors.

Figure 1
Figure 1. Overview of different modalities in surgical video and applications. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Execution flow of the proposed methodology. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. General scheme of the frame caption generation module. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visual walkthrough of the proposed workflow applied to a sample surgical video clip. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Example of frame and clip caption generation from annotated surgical video. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Reliability diagrams showing Expected Calibration Error before (left) and after (right) temperature scaling. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Mean Average Precision (mAP) comparison across three methods: SurgT, CLIP Surgical, and our proposed [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 45 canonical work pages

  1. [1]

    Gaze-assisted automatic captioning of fetal ultrasound videos using three-way multi-modal deep neural networks

    Mohammad Alsharid, Yifan Cai, Harshita Sharma, Lior Drukker, Aris T Papageorghiou, and J Alison Noble. Gaze-assisted automatic captioning of fetal ultrasound videos using three-way multi-modal deep neural networks. Medical Image Analysis, 82:102630, 2022. 16 Surgery video report generation PRIME AI paper

    work page 2022

  2. [2]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6077–6086, 2018

    2018

  3. [3]

    Croma: Cross-modal attention for visual question answering in robotic surgery

    Greetta Antonio, Jobin Jose, Sudhish N George, and Kiran Raja. Croma: Cross-modal attention for visual question answering in robotic surgery. In International Conference on Pattern Recognition, pages 459–471. Springer, 2024

    work page 2024

  4. [4]

    Vivit: A video vision transformer

    Anurag Arnab et al. Vivit: A video vision transformer. ICCV, 2021

    work page 2021

  5. [5]

    Video-based coaching in surgical education: a systematic review and meta-analysis

    Knut Magne Augestad, Khayam Butt, Dejan Ignjatovic, Deborah S Keller, and Ravi Kiran. Video-based coaching in surgical education: a systematic review and meta-analysis. Surgical endoscopy, 34:521–535, 2020

    work page 2020

  6. [6]

    M3d: Advancing 3d medical image analysis with multi-modal large language models

    Fan Bai, Yuxin Du, Tiejun Huang, Max Q-H Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models. arXiv preprint arXiv:2404.00578, 2024

    arXiv 2024

  7. [7]

    Space-time attention networks for video understanding

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Space-time attention networks for video understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 937–947, 2021

    work page 2021

  8. [8]

    Language models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020

    work page 1901

Show all 70 references
  1. [9]

    Cholect50: A dataset for surgical video understanding, 2020

    CAMMA. Cholect50: A dataset for surgical video understanding, 2020. URL https://github.com/ CAMMA-public/cholect50. Accessed: 2025-03-08

    2020

  2. [10]

    Unmasking deception: a topic-oriented multimodal approach to uncover false information on social media

    Riccardo Cantini, Cristian Cosentino, Irene Kilanioti, Fabrizio Marozzo, and Domenico Talia. Unmasking deception: a topic-oriented multimodal approach to uncover false information on social media. Machine Learning, 114(1):13, 2025. doi: 10.1007/s10994-024-06727-4. URL https://...

    2025 doi

  3. [11]

    Harnessing prompt- based large language models for disaster monitoring and automated reporting from social media feedback

    Riccardo Cantini, Cristian Cosentino, Fabrizio Marozzo, Domenico Talia, and Paolo Trunfio. Harnessing prompt- based large language models for disaster monitoring and automated reporting from social media feedback. Online Social Networks and Media, 2025. URL https://api.semanti...

    2025

  4. [12]

    Surgical video captioning with mutual-modal concept alignment

    Zhen Chen, Qingyu Guo, Leo KT Yeung, Danny TM Chan, Zhen Lei, Hongbin Liu, and Jinqiao Wang. Surgical video captioning with mutual-modal concept alignment. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 24–34. Springer, 2023

    2023

  5. [13]

    Vision language models in medicine

    Beria Chingnabe Kalpelbe, Angel Gabriel Adaambiik, and Wei Peng. Vision language models in medicine. arXiv e-prints, pages arXiv–2503, 2025

    2025

  6. [14]

    Meshed-memory transformer for image captioning

    Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10578–10587, 2020

    2020

  7. [15]

    Attribution-noncommercial-sharealike 4.0 international (cc by-nc-sa 4.0)

    Creative Commons. Attribution-noncommercial-sharealike 4.0 international (cc by-nc-sa 4.0). https: //creativecommons.org/licenses/by-nc-sa/4.0/ , 2013. Accessed: 2025-04-18

    2013

  8. [16]

    Class-balanced loss based on effective number of samples

    Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9268–9277, 2019

    2019

  9. [17]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  10. [18]

    Surgical video analysis: an emerging tool for improving surgeon performance, 2015

    Justin B Dimick and Oliver A Varban. Surgical video analysis: an emerging tool for improving surgeon performance, 2015

    2015

  11. [20]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021 arXiv

  12. [21]

    Videoorion: Tokenizing object dynamics in videos, 2024

    Yicheng Feng, Yijiang Li, Wanpeng Zhang, Sipeng Zheng, and Zongqing Lu. Videoorion: Tokenizing object dynamics in videos, 2024. URL https://arxiv.org/abs/2411.16156. 17 Surgery video report generation PRIME AI paper

    2024 arXiv

  13. [22]

    Image-text surgery: Efficient concept learning in image captioning by generating pseudopairs

    Kun Fu, Jin Li, Junqi Jin, and Changshui Zhang. Image-text surgery: Efficient concept learning in image captioning by generating pseudopairs. IEEE transactions on neural networks and learning systems , 29(12): 5910–5921, 2018

    2018

  14. [23]

    Using surgical video to improve technique and skill

    Tyler R Grenda, Jason C Pradarelli, and Justin B Dimick. Using surgical video to improve technique and skill. Annals of surgery, 264(1):32–33, 2016

    2016

  15. [24]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1321–1330, 2017

    2017

  16. [25]

    Hashimoto, Guy Rosman, Daniela Rus, and Ozanan R

    Daniel A. Hashimoto, Guy Rosman, Daniela Rus, and Ozanan R. Meireles. Artificial intelligence in surgery: Promises and perils. Annals of Surgery, 268(1):70–76, July 2018. doi: 10.1097/SLA.0000000000002693. URL https://pmc.ncbi.nlm.nih.gov/articles/PMC5995666/

    2018 doi

  17. [26]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  18. [27]

    What do we need to build explainable ai systems for the medical domain? arXiv preprint arXiv:1712.09923, 2017

    Andreas Holzinger, Chris Biemann, Constantinos S Pattichis, and Douglas B Kell. What do we need to build explainable ai systems for the medical domain? arXiv preprint arXiv:1712.09923, 2017

    2017 arXiv

  19. [28]

    Advancing medical imaging with language models: featuring a spotlight on chatgpt

    Mingzhe Hu, Joshua Qian, Shaoyan Pan, Yuheng Li, Richard LJ Qiu, and Xiaofeng Yang. Advancing medical imaging with language models: featuring a spotlight on chatgpt. Physics in Medicine & Biology, 69(10):10TR01, 2024

    2024

  20. [29]

    Exploring video captioning techniques: A comprehensive survey on deep learning methods

    Saiful Islam, Aurpan Dash, Ashek Seum, Amir Hossain Raj, Tonmoy Hossain, and Faisal Muhammad Shah. Exploring video captioning techniques: A comprehensive survey on deep learning methods. SN Computer Science, 2(2):1–28, 2021

    2021

  21. [30]

    Multi-task recurrent convolutional network with correlation loss for surgical video analysis

    Yueming Jin, Huaxia Li, Qi Dou, Hao Chen, Jing Qin, Chi-Wing Fu, and Pheng-Ann Heng. Multi-task recurrent convolutional network with correlation loss for surgical video analysis. Medical image analysis, 59:101572, 2020

    2020

  22. [31]

    Yuyi et al. Jin. Surgt: Transformer for surgical action triplet recognition. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 388–397. Springer, 2020

    2020

  23. [32]

    Predicting decompression surgery by applying multimodal deep learning to patients’ structured and unstructured health data

    Chethan Jujjavarapu, Pradeep Suri, Vikas Pejaver, Janna Friedly, Laura S Gold, Eric Meier, Trevor Cohen, Sean D Mooney, Patrick J Heagerty, and Jeffrey G Jarvik. Predicting decompression surgery by applying multimodal deep learning to patients’ structured and unstructured heal...

    2023

  24. [33]

    Stvs: Spatio-temporal feature fusion for video summarization

    Shamal Kashid, Lalit K Awasthi, Krishan Berwal, and Parul Saini. Stvs: Spatio-temporal feature fusion for video summarization. IEEE MultiMedia, 2024

    2024

  25. [34]

    Intraoperative video analysis and machine learning models will change the future of surgical training

    Michal Kawka, Tamara MH Gall, Chihua Fang, Rong Liu, and Long R Jiao. Intraoperative video analysis and machine learning models will change the future of surgical training. Intelligent Surgery, 1:13–15, 2022

    2022

  26. [35]

    Video question-answering techniques, benchmark datasets and evaluation metrics leveraging video captioning: A comprehensive survey

    Khushboo Khurana and Umesh Deshpande. Video question-answering techniques, benchmark datasets and evaluation metrics leveraging video captioning: A comprehensive survey. IEEE Access, 9:43799–43823, 2021

    2021

  27. [36]

    Generating automatic surgical captions using a contrastive language-image pre-training model for nephrectomy surgery images

    Sevdenur Kütük, Tuba Ça ˘glıkantar, and Duygu Sarıkaya. Generating automatic surgical captions using a contrastive language-image pre-training model for nephrectomy surgery images. In 2024 32nd Signal Processing and Communications Applications Conference (SIU), pages 1–4. IEEE, 2024

    2024

  28. [37]

    Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training

    Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In AAAI Conference on Artificial Intelligence, volume 34, pages 11336–11344, 2020

    2020

  29. [38]

    Oscar: Object-semantics aligned pre-training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision (ECCV), pages 121...

    2020

  30. [39]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, 2004

    2004

  31. [40]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980–2988, 2017

    2017

  32. [41]

    A survey on deep learning in medical image analysis

    Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey on deep learning in medical image analysis. Medical image analysis, 42:60–88, 2017

    2017

  33. [42]

    Video content analysis of surgical procedures

    Constantinos Loukas. Video content analysis of surgical procedures. Surgical endoscopy, 32:553–568, 2018. 18 Surgery video report generation PRIME AI paper

    2018

  34. [43]

    Vilbert: Pretraining task-agnostic visiolinguistic representa- tions for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representa- tions for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019

    1908 arXiv

  35. [44]

    BioGPT: generative pre-trained transformer for biomedical text generation and mining

    Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6), 09 2022. ISSN 1477-4054. doi: 10.1093/bib/bbac409. URL https://doi....

    2022 doi

  36. [45]

    Surgical data science for next- generation interventions

    Lena Maier-Hein, Swaroop S Vedula, Stefanie Speidel, Nassir Navab, Ron Kikinis, Adrian Park, Matthias Eisenmann, Hubertus Feussner, Germain Forestier, Stamatia Giannarou, et al. Surgical data science for next- generation interventions. Nature Biomedical Engineering, 1(9):691–696, 2017

    2017

  37. [46]

    Is online video-based education an effective method to teach basic surgical skills to students and surgical trainees? a systematic review and meta-analysis

    Brooke Perrin Mao, Makayla L Teichroeb, Taina Lee, Germaine Wong, Tony Pang, and Henry Pleass. Is online video-based education an effective method to teach basic surgical skills to students and surgical trainees? a systematic review and meta-analysis. Journal of surgical educa...

    2022

  38. [47]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. URL https://openai.com/research/gpt-4

    2023

  39. [48]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022

    2022 arXiv

  40. [49]

    Bleu: A method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  41. [50]

    Dense video captioning: A survey of techniques, datasets and evaluation protocols

    Iqra Qasim, Alexander Horsch, and Dilip Prasad. Dense video captioning: A survey of techniques, datasets and evaluation protocols. ACM Computing Surveys, 57(6):1–36, 2025

    2025

  42. [51]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019

    2019

  43. [52]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel et al. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020

    2020

  44. [53]

    Dl4burn: Burn surgical candidacy prediction using multimodal deep learning

    Sirisha Rambhatla, Samantha Huang, Loc Trinh, Mengfei Zhang, Boyuan Long, Mingtao Dong, Vyom Unadkat, Haig A Yenikomshian, Justin Gillenwater, and Yan Liu. Dl4burn: Burn surgical candidacy prediction using multimodal deep learning. In AMIA Annual Symposium Proceedings, volume ...

    2021

  45. [54]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108, 2019

    1910 arXiv

  46. [55]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS 2019, 2019

    2019

  47. [56]

    Evolution of visual data captioning methods, datasets, and evaluation metrics: A comprehensive survey

    Dhruv Sharma, Chhavi Dhiman, and Dinesh Kumar. Evolution of visual data captioning methods, datasets, and evaluation metrics: A comprehensive survey. Expert Systems with Applications, 221:119773, 2023

    2023

  48. [57]

    Sara Mahdavi, Joelle Barral, Dale Webster, Greg S

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y ...

    2023 arXiv

  49. [58]

    Automated radiology report generation: A review of recent advances

    Phillip Sloan, Philip Clatworthy, Edwin Simpson, and Majid Mirmehdi. Automated radiology report generation: A review of recent advances. IEEE Reviews in Biomedical Engineering, 2024

    2024

  50. [59]

    The role of large language models in medical image processing: a narrative review

    Dianzhe Tian, Shitao Jiang, Lei Zhang, Xin Lu, and Yiyao Xu. The role of large language models in medical image processing: a narrative review. Quantitative Imaging in Medicine and Surgery, 14(1):1108, 2023

    2023

  51. [60]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML), pages 10347–10357, 2021

    2021

  52. [61]

    On large visual language models for medical imaging analysis: An empirical study

    Minh-Hao Van, Prateek Verma, and Xintao Wu. On large visual language models for medical imaging analysis: An empirical study. In 2024 IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), pages 172–176. IEEE, 2024

    2024

  53. [62]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), pages 5998–6008, 2017. 19 Surgery video report generation PRI...

    2017

  54. [63]

    A novel multimodal deep learning model for preoperative prediction of microvascular invasion and outcome in hepatocellular carcinoma

    Fang Wang, Qingqing Chen, Yinan Chen, Yajing Zhu, Yuanyuan Zhang, Dan Cao, Wei Zhou, Xiao Liang, Yunjun Yang, Lanfen Lin, et al. A novel multimodal deep learning model for preoperative prediction of microvascular invasion and outcome in hepatocellular carcinoma. European Journ...

    2023

  55. [64]

    Endochat: Grounded multimodal large language model for endoscopic surgery

    Guankun Wang, Long Bai, Junyi Wang, Kun Yuan, Zhen Li, Tianxu Jiang, Xiting He, Jinlin Wu, Zhen Chen, Zhen Lei, et al. Endochat: Grounded multimodal large language model for endoscopic surgery. arXiv preprint arXiv:2501.11347, 2025

    2025 arXiv

  56. [65]

    Chatcad: Interactive computer-aided diagnosis on medical image using large language models

    Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. Chatcad: Interactive computer-aided diagnosis on medical image using large language models. arXiv preprint arXiv:2302.07257, 2023

    2023 arXiv

  57. [66]

    Learning domain adaptation with model calibration for surgical report generation in robotic surgery

    Mengya Xu, Mobarakol Islam, Chwee Ming Lim, and Hongliang Ren. Learning domain adaptation with model calibration for surgical report generation in robotic surgery. In 2021 IEEE international conference on robotics and automation (ICRA), pages 12350–12356. IEEE, 2021

    2021

  58. [67]

    Benchmarking large language models on summarization tasks

    Jiachen Zhang, Jianyu Wang, Pengfei Liu, Ming Shi, Yelong Huang, Qi Zhang, and Xiaodong He. Benchmarking large language models on summarization tasks. arXiv preprint arXiv:2301.13848, 2023

    2023 arXiv

  59. [68]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations (ICLR), 2020

    2020

  60. [69]

    Dilated temporal relational adversarial network for generic video summarization

    Yujia Zhang, Michael Kampffmeyer, Xiaodan Liang, Dingwen Zhang, Min Tan, and Eric P Xing. Dilated temporal relational adversarial network for generic video summarization. Multimedia Tools and Applications, 78: 35237–35261, 2019

    2019

  61. [70]

    Dense video captioning using graph-based sentence summarization

    Zhiwang Zhang, Dong Xu, Wanli Ouyang, and Luping Zhou. Dense video captioning using graph-based sentence summarization. IEEE Transactions on Multimedia, 23:1799–1810, 2020

    2020

  62. [71]

    Surgical activity recognition in robot-assisted radical prostatectomy using deep learning

    Aneeq Zia, Andrew Hung, Irfan Essa, and Anthony Jarc. Surgical activity recognition in robot-assisted radical prostatectomy using deep learning. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September ...

    2018

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.