Pith. sign in

REVIEW 5 major objections 5 minor 4 cited by

A surgery-specific multimodal LLM beats general video models on temporal questions, reporting 60.9% time-spot accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A surgical-video multimodal LLM with instrument-centric masked pretraining and interleaved temporal embeddings beats general video LLMs on a new CholecT50-derived benchmark.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection SurgLLM is a coherent, well-ablated MLLM for surgical video, but the reported margins over baselines rest on a pseudo-label loop and a GPT-4 judge, so the numbers are conditional until the authors release data and tighten the protocol. the 5 major comments →

arxiv 2509.00357 v1 pith:MQFAKTEI submitted 2025-08-30 cs.CV cs.AIcs.LG

SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding

Zhen Chen , Xingjian Luo , Kun Yuan , Jinlin Wu , Danny T.M. Chan , Nassir Navab , Hongbin Liu , Zhen Lei
show 1 more author
Jiebo Luo
This is my paper
classification cs.CV cs.AIcs.LG
keywords surgical video understandingmultimodal large language modelmasked video reconstructiontemporal visual question answeringinstrument-centric maskingtask dynamic ensembleCholecT50surgical captioning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Surgeons rely on endoscopic video, but general-purpose video AI models were trained on natural scenes and cope poorly with surgery's visual character: small moving instruments against a relatively static background, with long stretches of near-redundant frames. SurgLLM is the paper's attempt to close that gap with one adaptable multimodal system. The paper claims that three targeted changes—instrument-centric masked video reconstruction during pretraining, interleaved timestamp-descriptor and vision embeddings during tuning, and a task router that selects per-question specialist parameters—let a single model outperform seven general video LLMs on surgical captioning, general VQA, and temporal VQA. The sharpest reported gain is on temporal questions: 60.9% accuracy on time-spot queries and 42.6% duration IoU, versus 42.2% and 23.3% for the best general baseline.

Core claim

The paper claims that the reason video LLMs struggle on surgery is not model capacity but mismatch: visual encoders pretrained on natural scenes miss instrument-centric foreground dynamics, and LLMs attach timestamps too loosely to the frames they describe. SurgLLM is a domain-adapted multimodal LLM built on four components: Surg-Pretrain (instrument-centric multi-scale Masked Video Reconstruction plus surgical video-text contrastive alignment), TM-Tuning (interleaving each video clip's temporal descriptor directly before its visual tokens), and a Surgical Task Dynamic Ensemble (a multi-task Q-Former with task-specific learnable memories, plus task-specific LoRA weights on the LLM, selected

What carries the argument

Three mechanisms carry the argument. (1) Instrument-centric Multi-scale Masked Video Reconstruction (MV-Recon): the input video is divided into 3D tubes at four temporal scales (2/4/8/16 frames); tubes whose reference frame contains an instrument bounding box are masked, with only a random 10% kept as hints, and a video decoder must reconstruct the masked tubes. This pushes the encoder to represent the moving instruments—the foreground that matters—rather than the repetitive background. (2) TM-Tuning builds the LLM input as interleaved sequences [S1, Hv1, S2, Hv2, ..., q], where each Si is a text descriptor such as 'this clip spans from i×t to (i+1)×t seconds' placed immediately before that

Load-bearing premise

All instrument-location labels that drive both pretraining (the MV-Recon masks) and the location/time benchmark come from a detector trained on only five annotated videos; if that detector does not transfer to the other 45 videos, both the pretraining signal and the evaluation labels are unreliable, and the reported gains could be an artifact.

What would settle it

Re-run the temporal VQA benchmark on a test set where instrument bounding boxes and event timestamps are manually annotated by surgeons rather than generated from the 5-video-trained detector, and compare SurgLLM against the same general baselines. If the 60.9% time-spot accuracy and 42.6% duration IoU collapse toward or below the baselines, the central claim that instrument-centric masking and interleaved temporal embeddings cause the gains is undermined—the apparent advantage would instead reflect shared label statistics between pretraining, QA generation, and evaluation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A single SurgLLM-style model can replace separate specialist tools for phase recognition, triplet detection, location, relation, movement, captioning, and temporal reasoning, making a unified computer-assisted-surgery assistant feasible on one video encoder and LLM backbone.
  • Time-awareness at 60.9% spot accuracy and 42.6% duration IoU makes intraoperative applications—alerting when an instrument enters a critical region, predicting phase changes, logging instrument usage times—candidates for real-time deployment, pending clinical validation.
  • Task-routed LoRA weights plus task-specific memories mean a new surgical task can be added by inserting one new memory/LoRA pair and its routing classifier, without re-tuning shared parameters or losing previous task performance.
  • MV-Recon's instrument-centric masking offers a transferable recipe for long videos: mask the sparse moving foreground rather than random patches, so repetitive background frames do not drown out the rare decisive moments.
  • The two-stage progressive optimization (surgical visual pretraining, then task-adaptive tuning) provides a concrete template for adapting general video LLMs to other specialized visual domains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark's QA pairs were generated by GPT-4 from the same triplet, phase, and location annotations the model is trained to reproduce, part of the reported temporal gap may come from the model learning label statistics rather than genuinely watching the video; a human-annotated temporal QA set on unseen procedure types would isolate this.
  • The interleaved time-descriptor design suggests a cheap extension to continuous time: instead of discrete clip descriptors, the LLM could receive per-frame or per-token timestamps, potentially supporting sub-second event localization beyond the tested 2-to-16-second granularity.
  • If MV-Recon transfers, the same multi-scale instrument-centric masking could be applied to other tool-centric endoscopic or industrial inspection videos with small foreground objects on near-static backgrounds; that is a testable claim beyond the paper's cholecystectomy experiments.
  • The task dynamic ensemble routes purely on query text; a natural stress test is whether the router can handle open-set questions (new phrasings, combined tasks) without collapsing to a single routine, which the paper does not evaluate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SurgLLM, a large multimodal model for surgical video understanding. Three contributions are presented: (i) Surgical Context-aware Multimodal Pretraining (Surg-Pretrain), which uses instrument-centric masked video reconstruction (MV-Recon) with multi-scale tube masking (Eq. 1-2) followed by video-text contrastive alignment; (ii) Temporal-aware Multimodal Tuning (TM-Tuning), which interleaves temporal descriptors with visual segment embeddings (Eq. 6); and (iii) a Surgical Task Dynamic Ensemble that routes queries to task-specific Q-Former memories and LoRA weights (Eq. 7-9). The authors construct a new benchmark from CholecT50 by using an instrument detector trained on CholecT50-Challenge to generate bounding-box pseudo-labels for all 50 videos, then using these boxes and CholecT50 annotations to prompt GPT-4 for captions and VQA data. Experiments compare SurgLLM against general video LLMs (VideoLLaMA, LLaMA-VID, Video-LLaVA, VideoLLaMA-v2, VideoChat2, Qwen2.5-VL) on captioning, general VQA, and temporal VQA, reporting large gains, e.g., 60.9% vs 42.2% time-spot accuracy and 42.6% vs 23.3% duration IoU.

Significance. If the reported results are substantiated, SurgLLM would be a meaningful step toward adapting video MLLMs to surgical domains, with a sensible focus on instrument-centric masking and interleaved temporal embeddings. The paper includes ablations for most components and a hyperparameter analysis. However, the strength of the empirical claims is currently limited by three protocol issues: unvalidated pseudo-labels that are load-bearing in both pretraining and evaluation, an underspecified baseline comparison that may be zero-shot, and a GPT-4-as-judge evaluation without human validation. These issues are addressable, so the underlying ideas have promise, but the evidence as presented does not yet support the 'significant improvements over SOTA' claim at face value.

major comments (5)
  1. [Section 4.1, Fig. 5, Eq. (1)-(2)] The instrument detector is trained on the 5 CholecT50-Challenge videos and applied to all 50 CholecT50 videos to produce bounding-box pseudo-labels. These boxes drive both the MV-Recon masks in Eq. (1)-(2) and the Location/Movement ground truth and GPT-4 caption prompts. The paper reports no mAP/IoU for the detector and no quantitative validation of the pseudo-labels; the statement 'manual filtering and automatically validate' is unspecified. If the detector transfers poorly, location/movement/temporal results are measured against incorrect boxes, and SurgLLM is uniquely advantaged because its pretraining and evaluation labels share the same detector bias. Please report detector performance on a held-out set with human-annotated boxes, describe the filtering/validation process, and ideally re-run the location/movement experiments on a subset with verified boxes.
  2. [Section 4.2, Tables 1 and Fig. 8] The comparison protocol is underspecified. It is unclear whether the baseline video LLMs were fine-tuned on the same 80% training split or evaluated zero-shot. If they were zero-shot, the comparison is not apples-to-apples because SurgLLM is fine-tuned on the benchmark (including the GPT-4-generated question formats and temporal QA style). This would explain the large margins in temporal VQA. Please state explicitly the training protocol for each baseline; if zero-shot, include at least one strong baseline fine-tuned on the same data (e.g., Video-LLaVA or Qwen2.5-VL with LoRA) to isolate the contribution of the proposed architectural components.
  3. [Section 4.1, 'Evaluation Metrics'] General VQA accuracy is computed by GPT-4 judging whether the model's answer matches the ground truth. No human validation of these judgments is reported, and no error bars or confidence intervals are given. Since the questions and answers were themselves generated by GPT-4 from the same annotations, the evaluation is partly circular: SurgLLM is fine-tuned to mimic GPT-4's phrasing, and GPT-4 then judges its outputs. Please provide (i) a human-validated subset (e.g., 200-300 samples) with agreement rate between GPT-4 and human judgments, and (ii) per-task accuracy with standard deviations across multiple runs or bootstrapped confidence intervals.
  4. [Section 4.1, Temporal VQA metrics] The time-spot accuracy is defined only as 'accuracy of time spot prediction with the ground truth.' It is unclear what tolerance is allowed (exact second? +/-1 second? within the same clip?). This matters because the reported margins (60.9 vs 42.2) are central to the temporal-awareness claim. Please define the matching criterion precisely and, if possible, evaluate also with a tolerance curve. Additionally, the ground-truth time spots are derived from the unvalidated pseudo-labels, so a human-verified subset is essential.
  5. [Table 2(b), ablation] The ablation table's first row reports accuracy without MV-Recon, without Task Dynamic Ensemble, and without multi-task Q-Former, yet it still achieves 56.8% average general VQA. What is the architecture in this row? A Q-Former is necessary to map visual tokens to the LLM, so the row likely uses a standard Q-Former; please clarify. The column labeling should distinguish 'multi-task Q-Former with shared memories' from 'no multi-task Q-Former (standard Q-Former)' to make the ablations interpretable.
minor comments (5)
  1. [Fig. 1 caption] Typo: 'Surg-Petrain' should be 'Surg-Pretrain'.
  2. [Section 3.3] Typo: 'textural-visual temporal interleave embeddings' should be 'textual-visual'.
  3. [Section 4.2.3] The comparison only includes general video LLMs. If surgical-domain MLLMs or surgical VQA models exist (e.g., SurgicalGPT, SurgVLP, Surgical-VQA), they should be discussed or compared to support the 'state-of-the-art' claim in the surgical context.
  4. [Fig. 8] Bar charts lack error bars and numeric labels on every bar; adding error bars and values would improve readability and support the robustness claims.
  5. [Section 3.2.1 and 4.4] The hint retention ratio r, multi-scale tube durations, and LoRA rank/alpha are hyperparameters, but only video segment length is analyzed. A brief sensitivity analysis for r and LoRA rank would strengthen the hyperparameter section.

Circularity Check

2 steps flagged

Instrument-detector pseudo-labels define both MV-Recon masks and Location/Movement ground truth; GPT-4 both authors and judges general-VQA, so reported gains are partly circular.

specific steps
  1. fitted input called prediction [Section 4.1 (Datasets) and Section 3.2.1, Eq. (1)-(2); Fig. 5]
    "we first train a surgical instrument detection model on the CholecT50-Challenge dataset, and then conduct the inference on the CholecT50 dataset to generate bounding box annotations of surgical instruments across all 50 videos. We perform the manual filtering and automatically validate these generated annotations using existing triplet annotations to ensure the accuracy of the information."

    These detector pseudo-labels are load-bearing twice. First, Eq. (1)-(2) define the MV-Recon instrument mask indicator from the same bounding boxes ('if Ti contains instruments'), so SurgLLM's video encoder is pretrained to center on detector-defined instrument regions. Second, the Location and Movement VQA questions and answers are generated from the same {locations} box lists (Figs. 20 and 22), so the evaluation labels are not independent of the detector. No detector mAP/IoU is reported, and the 'manual filtering and automatically validate' step is unspecified. The reported Location (61.4%) and Movement (64.4%) gains can therefore reflect shared detector bias rather than independent surgical understanding.

  2. other [Section 4.1 (dataset generation and Evaluation Metrics); Fig. 5]
    "GPT-4-guided VQA generation ... We generate dense captions using GPT-4 [75] ... For the general VQA tasks, we employ GPT-4 [75] to validate the correctness of the prediction given the ground truth"

    The same GPT-4 system is used to author the training QA/caption data and to judge whether SurgLLM's general-VQA answers are correct. SurgLLM is instruction-tuned on GPT-4-written QA pairs, so high accuracy on the GPT-4 judge can reflect stylistic mimicry of the teacher/judge rather than independent surgical understanding. This creates a self-referential evaluation loop for the general-VQA numbers, although the captioning metrics and temporal IoU/time-spot accuracy are computed objectively.

full rationale

The core architecture is not circular: MV-Recon is self-supervised reconstruction (Eqs. 3-4), the contrastive alignment uses standard VTC/VTM/MLM objectives, and the temporal interleaving and task ensemble are architectural choices supported by ablations. Captioning metrics (BLEU/CIDEr/ROUGE-L/METEOR) and temporal IoU/time-spot accuracy provide partially independent evidence. However, the Location and Movement VQA tasks are not independent: the instrument detector trained on 5 challenge videos produces bounding-box pseudo-labels that both define the MV-Recon instrument mask (Eq. 1-2) and supply the {locations} lists from which GPT-4 writes the Location/Movement questions and answers. SurgLLM is therefore pretrained and evaluated on the same detector outputs, with no reported detector mAP/IoU or external validation of the pseudo-labels. Additionally, GPT-4 both generates the VQA/caption data and judges general-VQA correctness, making that evaluation partially self-referential. These issues affect only a subset of tasks and do not collapse the whole derivation, but they undermine the strongest claim of significant improvements on location/movement and general VQA. No load-bearing self-citation chain is present.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claims rest on the validity of CholecT50-derived pseudo-annotations and GPT-4-generated benchmark data, plus several hand-chosen hyperparameters. The model itself introduces no new physical or conceptual entities beyond standard neural network components (task-specific memories, LoRA adapters), which have internal empirical support.

free parameters (4)
  • hint retention ratio r = 10%
    Hyperparameter in Eq. (2): proportion of instrument-containing tubes kept as hints for MV-Recon. Chosen by hand without sensitivity analysis.
  • multi-scale tube durations = {2, 4, 8, 16} frames
    Temporal scales of tube masking in Surg-Pretrain; selected by hand, no ablation per scale is reported.
  • video segment duration t = 4 seconds
    Length of interleaved temporal clips in TM-Tuning; selected post hoc from the test-set analysis in Table 4, which is a form of tuning on the test split.
  • LoRA rank and alpha = 8 and 8
    Rank and alpha for task-specific LoRA adapters in the dynamic ensemble; chosen by hand.
axioms (5)
  • domain assumption CholecT50 phase and triplet annotations are accurate
    Ground truth for the benchmark is derived from CholecT50 annotations; any errors propagate to all captions, VQA labels, and final evaluation.
  • domain assumption The detector trained on CholecT50-Challenge generalizes to CholecT50
    Pseudo-label bounding boxes from this detector are used both for MV-Recon masks and for benchmark location labels (Section 4.1, Fig. 5). No evidence of generalization quality is given beyond manual filtering.
  • domain assumption GPT-4-generated captions and QA pairs are correct and free of style bias
    The entire training set and ground-truth answers are produced by GPT-4 from the same annotation lists; no error rate or human verification statistics are reported.
  • domain assumption GPT-4-as-judge reliably scores VQA correctness
    General VQA accuracy is computed by asking GPT-4 to validate predictions against ground truth; no human evaluation or agreement measure is provided.
  • standard math VideoMAE and Vicuna pretrained weights provide a suitable starting point
    The vision encoder and LLM are initialized from public pretrained checkpoints; this is a standard background assumption for MLLM research.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding." pith.science (2026). https://pith.science/paper/MQFAKTEI

@misc{pith2026250900357,
  author       = {Pith},
  title        = {Pith review of: SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MQFAKTEI}},
  note         = {Machine review of arXiv:2509.00357}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Surgical video understanding is crucial for facilitating Computer-Assisted Surgery (CAS) systems. Despite significant progress in existing studies, two major limitations persist, including inadequate visual content perception and insufficient temporal awareness in surgical videos, and hinder the development of versatile CAS solutions. In this work, we propose the SurgLLM framework, an effective large multimodal model tailored for versatile surgical video understanding tasks with enhanced spatial focus and temporal awareness. Specifically, to empower the spatial focus of surgical videos, we first devise Surgical Context-aware Multimodal Pretraining (Surg-Pretrain) for the video encoder of SurgLLM, by performing instrument-centric Masked Video Reconstruction (MV-Recon) and subsequent multimodal alignment. To incorporate surgical temporal knowledge into SurgLLM, we further propose Temporal-aware Multimodal Tuning (TM-Tuning) to enhance temporal reasoning with interleaved multimodal embeddings. Moreover, to accommodate various understanding tasks of surgical videos without conflicts, we devise a Surgical Task Dynamic Ensemble to efficiently triage a query with optimal learnable parameters in our SurgLLM. Extensive experiments performed on diverse surgical video understanding tasks, including captioning, general VQA, and temporal VQA, demonstrate significant improvements over the state-of-the-art approaches, validating the effectiveness of our SurgLLM in versatile surgical video understanding. The source code is available at https://github.com/franciszchen/SurgLLM.

Figures

Figures reproduced from arXiv: 2509.00357 by Danny T.M. Chan, Hongbin Liu, Jiebo Luo, Jinlin Wu, Kun Yuan, Nassir Navab, Xingjian Luo, Zhen Chen, Zhen Lei.

Figure 1
Figure 1. Figure 1: (a) The overview of the SurgLLM training pipeline, including Surgical Context-aware Multimodal Pretraining [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of Surg-Pretrain for SurgLLM. (a) Multi-scale Instrument-centric Masked Video Reconstruction adopts [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of Temporal-aware Multimodal Tuning (TM-Tuning) and the Surgical Task Dynamic Ensemble. (a) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of temporal embedding strategies for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The dataset construction pipeline for comprehensive [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: The radar chart comparison of our SurgLLM and [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparison of SurgLLM and state-of-the-art video LLMs across surgical VQA tasks. The general VQA tasks include [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Analysis of input demonstrates that surgical video [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Examples of caption generation in our processed CholecT50 dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Examples of general VQA and temporal VQA in our processed CholecT50 dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Comparison of SurgLLM and state-of-the-art Video LLMs on the caption generation task. A green part refers to a [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Comparison of SurgLLM and state-of-the-art Video LLMs on the general VQA regarding the location task. A green [PITH_FULL_IMAGE:figures/full_fig_p016_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Comparison of SurgLLM and state-of-the-art Video LLMs on the general VQA regarding the movement task. A [PITH_FULL_IMAGE:figures/full_fig_p016_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Comparison of SurgLLM and state-of-the-art Video LLMs on the general VQA regarding the phase task. A green [PITH_FULL_IMAGE:figures/full_fig_p017_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Comparison of SurgLLM and state-of-the-art Video LLMs on the general VQA regarding the relation task. A green [PITH_FULL_IMAGE:figures/full_fig_p017_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Comparison of SurgLLM and state-of-the-art Video LLMs on the general VQA regarding the triplet task. A green [PITH_FULL_IMAGE:figures/full_fig_p018_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Comparison of SurgLLM and state-of-the-art Video LLMs on the temporal VQA regarding the time spot task. A [PITH_FULL_IMAGE:figures/full_fig_p018_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: The prompt for caption generation in our processed CholecT50 dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: The prompt for general VQA regarding the phase task in our processed CholecT50 dataset. [PITH_FULL_IMAGE:figures/full_fig_p020_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: The prompt for temporal VQA regarding the duration task in our processed CholecT50 dataset. [PITH_FULL_IMAGE:figures/full_fig_p021_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: The prompt for temporal VQA regarding the time spot task in our processed CholecT50 dataset. [PITH_FULL_IMAGE:figures/full_fig_p022_22.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery

    cs.CV 2026-06 unverdicted novelty 7.0

    SurgAtlas is a new dataset of 15,291 surgical videos totaling 2,391 hours with multi-level annotations that supports finetuning models to competitive performance on surgical benchmarks.

  2. SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark

    cs.CV 2026-04 unverdicted novelty 7.0

    SurgCoT is a new benchmark that evaluates chain-of-thought spatiotemporal reasoning in multimodal large language models on surgical videos using five defined dimensions and an annotation protocol of Question-Option-Kn...

  3. MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding

    cs.CV 2025-12 conditional novelty 6.0

    MedGRPO applies cross-dataset reward normalization and a clinical LLM judge within multi-task RL to improve vision-language models on heterogeneous medical video understanding tasks using the new MedVidBench dataset.

  4. UniReason-Med: A Shared Grounded Reasoning Interface for 2D-to-3D Transfer in Medical VQA

    cs.CV 2026-06 unverdicted novelty 4.0

    UniReason-Med introduces a unified framework for 2D and 3D medical VQA with shared grounded reasoning, trained on a 220K dataset, claiming that joint 2D+3D supervision improves 3D performance over 3D-only training.

Reference graph

Works this paper leans on

79 extracted references · 57 canonical work pages · cited by 4 Pith papers · 1 internal anchor

  1. [1]

    Surgical data science for next-generation interventions

    Lena Maier-Hein, Swaroop S Vedula, Stefanie Speidel, Nassir Navab, Ron Kikinis, Adrian Park, Matthias Eisenmann, Hubertus Feussner, Germain Forestier, Stamatia Giannarou, et al. Surgical data science for next-generation interventions. Nature Biomedical Engineering, 1(9):691–696, 2017

    work page 2017

  2. [2]

    Artificial intelligence and automation in endoscopy and surgery

    François Chadebecq, Laurence B Lovat, and Danail Stoyanov. Artificial intelligence and automation in endoscopy and surgery. Nature Reviews Gastroenterology & Hepatology , 20(3):171–182, 2023

    work page 2023

  3. [3]

    Concepts and trends in autonomy for robot-assisted surgery

    Paolo Fiorini, Ken Y Goldberg, Yunhui Liu, and Russell H Taylor. Concepts and trends in autonomy for robot-assisted surgery. Proceedings of the IEEE , 110(7):993–1011, 2022

    work page 2022

  4. [4]

    Robot-assisted minimally invasive surgery—surgical robotics in the data age

    Tamas Haidegger, Stefanie Speidel, Danail Stoyanov, and Richard M Satava. Robot-assisted minimally invasive surgery—surgical robotics in the data age. Proceedings of the IEEE , 110(7):835–846, 2022

    work page 2022

  5. [5]

    Unified detection and tracking of instru- ments during retinal microsurgery

    Raphael Sznitman, Rogerio Richa, Russell H Taylor, Bruno Jedynak, and Gregory D Hager. Unified detection and tracking of instru- ments during retinal microsurgery. IEEE T ransactions on Pattern Analysis and Machine Intelligence , 35(5):1263–1273, 2012

    work page 2012

  6. [6]

    Probabilistic tracking of affine-invariant anisotropic regions

    Stamatia Giannarou, Marco Visentini-Scarzanella, and Guang- Zhong Yang. Probabilistic tracking of affine-invariant anisotropic regions. IEEE T ransactions on Pattern Analysis and Machine Intelli- gence, 35(1):130–143, 2012

    work page 2012

  7. [7]

    See- through vision with unsupervised scene occlusion reconstruction

    Samyakh Tukra, Hani J Marcus, and Stamatia Giannarou. See- through vision with unsupervised scene occlusion reconstruction. IEEE T ransactions on Pattern Analysis and Machine Intelligence , 44(7):3779–3790, 2021

    work page 2021

  8. [8]

    Surgicalsam: Efficient class promptable surgical instrument segmentation

    Wenxi Yue, Jing Zhang, Kun Hu, Yong Xia, Jiebo Luo, and Zhiyong Wang. Surgicalsam: Efficient class promptable surgical instrument segmentation. In AAAI, 2024

    work page 2024

  9. [9]

    Asi-seg: Audio- driven surgical instrument segmentation with surgeon intention understanding

    Zhen Chen, Zongming Zhang, Wenwu Guo, Xingjian Luo, Long Bai, Jinlin Wu, Hongliang Ren, and Hongbin Liu. Asi-seg: Audio- driven surgical instrument segmentation with surgeon intention understanding. In IROS, 2024

    work page 2024

  10. [11]

    Temporal memory relation network for workflow recognition from surgical video

    Yueming Jin, Yonghao Long, Cheng Chen, Zixu Zhao, Qi Dou, and Pheng-Ann Heng. Temporal memory relation network for workflow recognition from surgical video. IEEE T ransactions on Medical Imaging, 40(7):1911–1923, 2021

    work page 1911

  11. [12]

    Surgplan: Surgical phase localization network for phase recognition

    Xingjian Luo, You Pang, Zhen Chen, Jinlin Wu, Zongmin Zhang, Zhen Lei, and Hongbin Liu. Surgplan: Surgical phase localization network for phase recognition. In ISBI. IEEE, 2024

    work page 2024

  12. [13]

    Soh, and Yousuf M

    Junhuan Zhu, Jiebo Luo, Jonathan M. Soh, and Yousuf M. Khalifa. A computer vision-based approach to grade simulated cataract surgeries. Mach. Vis. Appl., 26(1):115–125, 2015

    work page 2015

  13. [14]

    Video-based surgical skill assessment using 3d convolutional neural networks

    Isabel Funke, Sören Torge Mees, Jürgen Weitz, and Stefanie Speidel. Video-based surgical skill assessment using 3d convolutional neural networks. International journal of computer assisted radiology and surgery, 14:1217–1225, 2019

    work page 2019

  14. [15]

    Towards unified surgical skill assessment

    Daochang Liu, Qiyue Li, Tingting Jiang, Yizhou Wang, Rulin Miao, Fei Shan, and Ziyu Li. Towards unified surgical skill assessment. In CVPR, pages 9522–9531, 2021

    work page 2021

  15. [16]

    Rethinking surgical captioning: End-to-end window-based mlp transformer using patches

    Mengya Xu, Mobarakol Islam, and Hongliang Ren. Rethinking surgical captioning: End-to-end window-based mlp transformer using patches. In MICCAI, pages 376–386. Springer, 2022

    work page 2022

  16. [17]

    Surgical video captioning with mutual-modal concept alignment

    Zhen Chen, Qingyu Guo, Leo KT Yeung, Danny TM Chan, Zhen Lei, Hongbin Liu, and Jinqiao Wang. Surgical video captioning with mutual-modal concept alignment. In MICCAI, pages 24–34. Springer, 2023

    work page 2023

  17. [18]

    Surgicalgpt: end-to-end language-vision gpt for visual question answering in surgery

    Lalithkumar Seenivasan, Mobarakol Islam, Gokul Kannan, and Hongliang Ren. Surgicalgpt: end-to-end language-vision gpt for visual question answering in surgery. In MICCAI, pages 281–290. Springer, 2023

    work page 2023

  18. [19]

    Surgical-vqla++: Adversarial contrastive learning for calibrated robust visual question-localized answering in robotic surgery

    Long Bai, Guankun Wang, Mobarakol Islam, Lalithkumar Seeni- vasan, An Wang, and Hongliang Ren. Surgical-vqla++: Adversarial contrastive learning for calibrated robust visual question-localized answering in robotic surgery. Information Fusion, page 102602, 2024

    work page 2024

  19. [20]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024

    2024

  20. [21]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742. PMLR, 2023

    work page 2023

  21. [22]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 35:23716–23736, 2022

    work page 2022

  22. [23]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023

    Pith/arXiv arXiv 2023

  23. [24]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P . Xing. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023

    work page 2023

  24. [25]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024

    Pith/arXiv arXiv 2024

  25. [26]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 , 2023

    Pith/arXiv arXiv 2023

  26. [27]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In NeurIPS, 2023

    work page 2023

  27. [28]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, volume 36, 2024

    work page 2024

  28. [29]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024

    Pith/arXiv arXiv 2024

  29. [30]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In CVPR, pages 13817–13827, 2024

    work page 2024

  30. [31]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR, pages 22195–22206, 2024

    work page 2024

  31. [32]

    Video-llama: An instruction- tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction- tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023

    Pith/arXiv arXiv 2023

  32. [33]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    Pith/arXiv arXiv 2025

  33. [34]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    Pith/arXiv arXiv 2024

  34. [35]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023

    Pith/arXiv arXiv 2023

  35. [36]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–

    work page

  36. [37]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. NeurIPS, 35:10078–10093, 2022

    work page 2022

  37. [38]

    Surgplan: Surgical phase localization network for phase recognition

    Xingjian Luo, You Pang, Zhen Chen, Jinlin Wu, Zongmin Zhang, Zhen Lei, and Hongbin Liu. Surgplan: Surgical phase localization network for phase recognition. In ISBI, pages 1–5. IEEE, 2024

    work page 2024

  38. [39]

    Surgical temporal action-aware network with sequence regularization for phase recognition

    Zhen Chen, Yuhao Zhai, Jun Zhang, and Jinqiao Wang. Surgical temporal action-aware network with sequence regularization for phase recognition. In BIBM, pages 1836–1841. IEEE, 2023

    work page 2023

  39. [40]

    Team-based surgical scheduling for improved patient access in a high-volume, tertiary head and neck cancer center

    Nicole C Schmitt, Martha Ryan, Tyler Halle, Amy Sherrod, J Trad Wadsworth, Mihir R Patel, and Mark W El-Deiry. Team-based surgical scheduling for improved patient access in a high-volume, tertiary head and neck cancer center. Annals of Surgical Oncology , 29(11):7002–7006, 2022. A VERSATILE LARGE MULTIMODAL MODEL WITH SPATIAL FOCUS AND TEMPORAL AWARENESS ...

    work page 2022

  40. [41]

    The loud surgeon behind the console: understanding team activities during robot- assisted surgery

    Judith Tiferes, Ahmed A Hussein, Ann Bisantz, Justen D Kozlowski, Mohamed A Sharif, Nathalie M Winder, Nabeeha Ahmad, Jenna Allers, Lora Cavuoto, and Khurshid A Guru. The loud surgeon behind the console: understanding team activities during robot- assisted surgery. Journal of surgical education , 73(3):504–512, 2016

    work page 2016

  41. [42]

    Internvideo: General video foundation models via generative and discriminative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191 , 2022

    Pith/arXiv arXiv 2022

  42. [43]

    Surgical data science: Emerging trends and future pathways

    Renáta Levendovics, Tamás Levendovics, Gernot Kronreif, and Tamás Haidegger. Surgical data science: Emerging trends and future pathways. Recent Advances in Intelligent Engineering: Volume Dedicated to Imre J. Rudas’ Seventy-Fifth Birthday , pages 65–84, 2024

    work page 2024

  43. [44]

    Artificial intelligence in surgery: the future is now

    Ahmad Guni, Piyush Varma, Joe Zhang, Matyas Fehervari, and Hutan Ashrafian. Artificial intelligence in surgery: the future is now. European Surgical Research, 65(1):22–39, 2024

    work page 2024

  44. [45]

    Vs-assistant: versatile surgery assistant on the demand of surgeons

    Zhen Chen, Xingjian Luo, Jinlin Wu, Danny Chan, Zhen Lei, Jinqiao Wang, Sebastien Ourselin, and Hongbin Liu. Vs-assistant: versatile surgery assistant on the demand of surgeons. arXiv preprint arXiv:2405.08272, 2024

    Pith/arXiv arXiv 2024

  45. [46]

    Endonet: a deep architecture for recognition tasks on laparoscopic videos

    Andru P Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE T ransactions on Medical Imaging, 36(1):86–97, 2016

    work page 2016

  46. [47]

    Rendezvous: Attention mechanisms for the recogni- tion of surgical action triplets in endoscopic videos

    Chinedu Innocent Nwoye, Tong Yu, Cristians Gonzalez, Barbara Seeliger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Rendezvous: Attention mechanisms for the recogni- tion of surgical action triplets in endoscopic videos. Medical Image Analysis, 78:102433, 2022

    work page 2022

  47. [48]

    2017 robotic instrument segmentation challenge

    Max Allan, A Shvets, Thomas Kurmann, Z Zhang, R Duggal, Y-H Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, Luis Garcia-Peraza-Herrera, W Li, Vladimir Iglovikov, H Luo, J Yang, Danail Stoyanov, L Maier-Hein, Stefanie Speidel, and M Azizian. 2017 robotic instrument segmentation challenge. 02 2019

    work page 2017

  48. [49]

    2018 robotic scene segmentation challenge

    Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, R Kadkhodamohammadi, I Luengo, Félix Fuentes, E Flouty, A Mo- hammed, M Pedersen, Avinash Kori, V Alex, G Krishnamurthi, David Rauber, Robert Mendel, Christoph Palm, Sophia Bano, Güinther Saibro, C Shih, and Stefanie Speidel. 2018 robotic scene segmentation challenge. 06 2021

    work page 2018

  49. [50]

    Surgical-vqa: Visual question answering in surgical scenes using transformer

    Lalithkumar Seenivasan, Mobarakol Islam, Adithya K Krishna, and Hongliang Ren. Surgical-vqa: Visual question answering in surgical scenes using transformer. In MICCAI, pages 33–43. Springer, 2022

    work page 2022

  50. [51]

    Advancing surgical vqa with scene graph knowledge

    Kun Yuan, Manasi Kattel, Joël L Lavanchy, Nassir Navab, Vinkle Srivastav, and Nicolas Padoy. Advancing surgical vqa with scene graph knowledge. International Journal of Computer Assisted Radiology and Surgery , pages 1–9, 2024

    work page 2024

  51. [52]

    Surgical activity triplet recognition via triplet disentanglement

    Yiliang Chen, Shengfeng He, Yueming Jin, and Jing Qin. Surgical activity triplet recognition via triplet disentanglement. In MICCAI, pages 451–461. Springer, 2023

    work page 2023

  52. [53]

    Rich feature hierarchies for accurate object detection and semantic segmentation

    Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pages 580–587, 2014

    work page 2014

  53. [54]

    Fast r-cnn

    Ross Girshick. Fast r-cnn. In ICCV, pages 1440–1448, 2015

    work page 2015

  54. [55]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, volume 28, 2015

    work page 2015

  55. [56]

    You only look once: Unified, real-time object detection

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, pages 779–788, 2016

    2016

  56. [57]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021

    work page 2021

  57. [58]

    nnu-net: a self-configuring method for deep learning- based biomedical image segmentation

    Fabian Isensee, Paul Jaeger, Simon Kohl, Jens Petersen, and Klaus Maier-Hein. nnu-net: a self-configuring method for deep learning- based biomedical image segmentation. Nature Methods, 18:1–9, 02 2021

    work page 2021

  58. [59]

    A real-time spatiotemporal ai model analyzes skill in open surgical videos

    Emmett D Goodman, Krishna K Patel, Yilun Zhang, William Locke, Chris J Kennedy, Rohan Mehrotra, Stephen Ren, Melody Y Guan, Maren Downing, Hao Wei Chen, et al. A real-time spatiotemporal ai model analyzes skill in open surgical videos. arXiv preprint arXiv:2112.07219, 2021

    Pith/arXiv arXiv 2021

  59. [60]

    Improving surgical techniques: Use of surgical procedures videos as learning tools-a multicentric study

    Ahmed Hassaan Malik, Shafaqat Ali, and Faraz Anwar Syed. Improving surgical techniques: Use of surgical procedures videos as learning tools-a multicentric study. Pakistan Armed Forces Medical Journal, 74(4):1110, 2024

    work page 2024

  60. [61]

    Augmenting Efficient Real-time Surgical Instrument Segmentation in Video with Point Tracking and Segment Anything

    Zijian Wu, Adam Schmidt, Peter Kazanzides, and Septimiu E Salcudean. Real-time surgical instrument segmentation in video using point tracking and segment anything. arXiv preprint arXiv:2403.08003, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  61. [62]

    Generative artificial intelligence in surgery

    Severin Rodler, Conner Ganjavi, Pieter De Backer, Vasileios Magou- lianitis, Lorenzo Storino Ramacciotti, Andre Luis De Castro Abreu, Inderbir S Gill, and Giovanni E Cacciamani. Generative artificial intelligence in surgery. Surgery, 2024

    work page 2024

  62. [63]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    2023

  63. [64]

    Video understanding with large language models: A survey.arXiv preprint arXiv:2312.17432, 2023

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. Video understanding with large language models: A survey.arXiv preprint arXiv:2312.17432, 2023

    arXiv 2023

  64. [65]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 , 2023

    Pith/arXiv arXiv 2023

  65. [66]

    Chatvideo: A tracklet-centric multimodal and versatile video understanding system

    Junke Wang, Dongdong Chen, Chong Luo, Xiyang Dai, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. Chatvideo: A tracklet-centric multimodal and versatile video understanding system. arXiv preprint arXiv:2304.14407, 2023

    Pith/arXiv arXiv 2023

  66. [67]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 , 2023

    Pith/arXiv arXiv 2023

  67. [68]

    Languagebind: Extending video-language pretraining to n-modality by language- based semantic alignment, 2023

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Wang HongFa, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Cai Wan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Languagebind: Extending video-language pretraining to n-modality by language- based semantic alignment, 2023

    work page 2023

  68. [69]

    Videollama 2: Advancing spatial-temporal mod- eling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal mod- eling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024

    Pith/arXiv arXiv 2024

  69. [70]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. In CVPR, pages 14271–14280, 2024

    work page 2024

  70. [71]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In CVPR, pages 14313–14323, 2024

    work page 2024

  71. [72]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR, pages 13700–13710, 2024

    work page 2024

  72. [73]

    Unmasked teacher: Towards training-efficient video foundation models

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, pages 19948–19960, 2023

    work page 2023

  73. [74]

    Gonzalez, and Nicolas Padoy

    Chinedu Innocent Nwoye, Tong Yu, Saurav Sharma, Aditya Murali, Deepak Alapatt, Cristians Vardazaryan, Armine ... Gonzalez, and Nicolas Padoy. Cholectriplet2022: Show me a tool and tell me the triplet: an endoscopic vision challenge for surgical action triplet detection. arXiv preprint arXiv:2204.14746 , 2023

    work page arXiv 2023

  74. [75]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    Pith/arXiv arXiv 2023

  75. [76]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017

    2017

  76. [77]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR, pages 14549– 14560, 2023

    work page 2023

  77. [78]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311–318, 2002

    2002

  78. [79]

    Cider: Consensus-based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, pages 4566–4575, 2015

    work page 2015

  79. [80]

    Calot's triangle dissection

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL Workshop, pages 65–72, 2005. A VERSATILE LARGE MULTIMODAL MODEL WITH SPATIAL FOCUS AND TEMPORAL AWARENESS FOR SURGICAL VIDEO UNDERSTANDING 14 Examples of CaptionGeneration Long CaptionIn the laparoscopic cholecystectomy s...

    work page 2005

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.