{"id":"9c0b92bb-3680-46a6-a436-5e7190409c52","arxiv_id":"2504.19918","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A modular vision-language pipeline for automatic surgical report generation from laparoscopic videos, evaluated on CholecT50 with high reported scores but a leakage-prone evaluation protocol.","lead":"This paper proposes a three-stage AI pipeline that detects surgical tools in video frames, writes captions for frames and clips, and asks GPT-4 to compose a full surgical report. The reported accuracy is high, but the evaluation is weakened by a frame-level data split and by caption targets that are generated from the same annotations the model receives as input.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Frame-level train/test split in §4.1 leaks near-duplicate frames from the same surgical videos, so reported detection and captioning scores do not establish generalization.","rationale":"I read the paper as claiming a working three-stage surgical documentation pipeline. For that claim to hold, the evaluation must measure generalization to unseen surgical videos. The frame-level split in §4.1 breaks that condition: because CholecT50 is 50 continuous videos at 1 fps and clips overlap by 16 frames, train and test contain near-duplicates from the same surgeries. This is not a stylistic concern; it directly inflates every reported metric, since the model can match test frames to near-identical training frames. The reader's weakest_assumption identifies exactly this issue, and I agree. The secondary circular-caption issue (ground truth built from the same annotations used as input) further weakens the summarization scores, but the split is the primary load-bearing flaw. I do not see a way to accept the central claim on the current evidence: re-evaluation with proper video-level splits and human-written or independently verified references would be required. Therefore the reader's REJECT verdict is appropriate, and I would leave it unchanged.","tokens_in":19573,"tokens_out":4130,"duration_ms":40720,"concrete_test":"Recompute the full evaluation with a video-level split: assign the 50 CholecT50 videos to 40 train / 5 validation / 5 test, keep all frames and clips of a given video in one split, and rerun object detection (Fig. 7 metrics) and the frame/clip captioning experiments (Tables 2 and 3). If test precision, mAP, or BERT F1 drops substantially or exhibits high variance across held-out videos, the reported numbers are an artifact of the frame-level split. As a diagnostic, also measure average pixel or feature similarity between nearest train/test frames; near-duplicate cross-split pairs are direct evidence of leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 partitions the dataset into 80% training and 10% test/validation frames after creating 32-frame clips with 16-frame overlap from continuous 1-fps videos. CholecT50 has only 50 videos, so adjacent frames and overlapping clips from the same procedure appear in both training and test. The model can memorize video-specific appearance (instrument lighting, anatomy, phase transitions) instead of learning general tool detection or action captioning. This invalidates the object-detection mAP in Fig. 7, the frame-caption metrics in Table 2, the clip-caption metrics in Table 3, and by extension the GPT-4 report assessment in §4.6, which consumes those leaked captions. The abstract's '96% precision / BERT 0.74' also do not match any table in the body, but the split is the more fundamental flaw. A secondary issue is that §4.1 states frame and clip captions are 'created artificially using the annotations,' so the reference text is a deterministic template built from the exact labels fed to the model, making BLEU/ROUGE/BERTScore partly circular. The temporal leak alone is sufficient to reject the paper as evidence for the claimed system.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a three-stage pipeline for automatic surgical documentation from laparoscopic videos: (1) multi-label object classification of instruments and targets from individual video frames using a Vision Transformer (ViT); (2) frame-level caption generation with a ViT/DistilBERT/T5 cross-attention model, followed by clip-level caption generation with a ViViT/FLAN-T5 model that fuses visual and textual features; and (3) synthesis of a full surgical report by prompting GPT-4 with the concatenated clip captions. The system is evaluated on CholecT50, and the authors report strong instrument/target recognition, frame- and clip-level captioning scores (BLEU, ROUGE, BERTScore), and an example GPT-4 generated report. The central claim is that the modular, temporally aware architecture is capable of accurate tool detection and coherent, clinically usable surgical report generation.","tokens_in":19739,"tokens_out":4369,"duration_ms":43469,"significance":"If the reported results were valid, the paper would make a useful engineering contribution to automated surgical documentation: it integrates established components (ViT, ViViT, T5, GPT-4) in a modular pipeline, publicly releases preprocessing scripts and trained models, and describes an error-aware training strategy that is a reasonable practical idea. However, the evaluation protocol has two fundamental flaws that invalidate the main empirical claims: a frame-level train/test split that leaks near-duplicate frames from the same surgical videos into both training and test sets, and captioning references that are deterministic templates generated from the very annotations the models receive as input. Because of these flaws, the reported detection, captioning, and summarization numbers cannot be interpreted as evidence of generalization or of semantic quality.","major_comments":[{"comment":"The train/test/validation split is performed at the frame level (80/10/10) rather than at the video level. CholecT50 consists of only 50 videos, the frames are extracted at 1 fps, and clips are created with 32 frames and a 16-frame overlap. Consequently, frames and overlapping clips from the same surgical video appear in both the training and test sets, and adjacent frames are near-duplicates. The model can memorize video-specific appearance, instrument lighting, and phase transitions instead of learning generalizable tool recognition or action captioning. This temporal leakage invalidates the object-detection mAP in Figure 7, the frame-caption metrics in Table 2, the clip-caption metrics in Table 3, and the qualitative GPT-4 report assessment in Section 4.6, all of which are central to the paper's claims.","section":"Section 4.1"},{"comment":"The frame and clip captions used as both training targets and evaluation references are 'created artificially using the annotations' (verbs, targets, phases), and the captioning models receive exactly those annotations as part of their input, as described in Sections 3.2 and 4.2.2. The reference captions are therefore deterministic templates built from the same information the model consumes. The BLEU, ROUGE, and BERTScore numbers thus measure how well the model reconstructs a fixed template, not how accurately or usefully it describes surgical content. An evaluation with human-authored references and a protocol that prevents the reference content from being fed as input is needed to support the captioning claims.","section":"Section 4.1"},{"comment":"The abstract reports '96% precision in tool detection' and 'BERT score of 0.74', but no table or figure in the body reports a precision of 0.96 or a BERTScore of 0.74. Figure 7 shows mean average precision values for instruments and targets, Table 2 lists BERT F1 up to 0.8052 for frame captions, and Table 3 reports BERT F1 0.7607 for the robust clip model. The headline numbers in the abstract are not traceable to any result shown in the experimental section, making the main advertised findings unverifiable from the manuscript.","section":"Abstract, Figure 7, Tables 2-3"}],"minor_comments":[{"comment":"The method described in Equations (2)-(4) is multi-label image classification, not object detection in the standard sense, since no bounding boxes or localizations are produced. The repeated use of 'object detection' and the mAP comparison in Figure 7 should be clarified or renamed to avoid ambiguity.","section":"Section 3.1"},{"comment":"The bar chart does not show numerical values on the bars, and the legend order is not aligned with the bar grouping, making it difficult to determine which bars correspond to which model and metric. Please add value labels and a clearer legend.","section":"Figure 7"},{"comment":"The sentence 'With a total of89827, the the clips are created...' contains typographical errors and a wrongly placed number, and the text should be revised for readability. Appendix A also contains the misspelling 'gallbloddger' in a predicted caption, which should be corrected.","section":"Section 4.1"}],"recommendation":"reject","confidential_remarks":"The frame-level split and the template-based caption references are load-bearing flaws: they undermine every quantitative result in the paper, and fixing them would require re-running all experiments with a video-level split and human-validated references. The mismatch between the abstract's headline numbers and the body is also a serious credibility issue for a journal submission. In my view this manuscript will need a substantially new evaluation campaign before it can be considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about this paper. The empirical claims are not backed by the evaluation. The train/test split in Section 4.1 is frame-level, so adjacent frames from the same 50 surgical videos are near-duplicates across train and test. That alone leaks video identity and inflates detection, caption, and summary numbers. And the caption references are not human text: Section 4.1 says frame and clip captions are \"created artificially using the annotations,\" i.e., deterministic templates built from the exact labels the model receives as input. So BLEU/BERTScore measure how well the model echoes a template, not what it understands about surgery.\n\nWhat the paper does well: it puts together a complete modular pipeline — ViT detection, frame captions with ViT+DistilBERT+T5, clip captions with ViViT, and GPT-4 report synthesis — and ships the code and trained models. The two-phase robust training, where each stage is trained on the previous stage's predictions, is a reasonable way to cope with error propagation, though the idea itself is not new. The calibration analysis (ECE before/after temperature scaling) is a nice touch, and the comparison against SurgT and CLIP Surgical is the right kind of thing to do.\n\nThe problems are not cosmetic. The leaked split means Figure 7 and Tables 2 and 3 don't tell you how the system will behave on new videos. The template references make the caption scores look far better than they are: the model is essentially given the triplet labels and asked to render a sentence, so a high BLEU is expected. On top of that, the abstract's \"96% precision and BERT 0.74\" appear nowhere in the body's tables, which is a mismatch the authors should fix. The final GPT-4 report is only qualitatively assessed, which is fine for a demo but not a claim of clinical usefulness.\n\nNovelty is low — every block is off-the-shelf and the integration is straightforward — but that alone wouldn't sink a solid systems paper. As written, though, the evaluation protocol sinks it.\n\nThis paper is for readers who want a blueprint for a surgical video summarization pipeline. They'll get that, but they should not trust the numbers. If the authors rerun the experiments with a video-level split (e.g., leave some videos out entirely) and evaluate captioning against human-written or at least non-circular references, this could be a decent contribution. As it stands, I would not accept it.\n\nMy recommendation for peer review: reject the current version, but with an invitation to revise. The topic is worth engaging, and the authors have done real engineering. They just haven't proven that it works.","headline":"The evaluation protocol sinks an otherwise competent pipeline: frame-level split leaks near-duplicate frames across train/test, and the caption references are templates built from the model's own input labels.","tokens_in":20340,"tokens_out":3193,"would_cite":false,"duration_ms":31742,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Surgical videos become structured reports through a three-stage transformer pipeline, with 96% tool-detection precision and BERT score 0.74.","keywords":["surgical video summarization","multimodal transformers","Vision Transformer","Video Vision Transformer","laparoscopic cholecystectomy","CholecT50","large language models","report generation"],"falsifier":"Take any held-out surgical video, run the pipeline, and compare tool detection precision on clips that share no frames with training, and the BERT score of clip captions against independently written surgeon reports instead of the templated captions; a substantial drop in either number would show the reported performance depends on frame leakage and template-matching rather than genuine generalization.","tokens_in":19299,"feed_emoji":"🏥","tokens_out":7107,"duration_ms":64676,"temperature":0.7,"pith_summary":"Automatic summarization of surgical videos matters for training, documentation, and post-operative review, and this paper proposes a three-stage pipeline to produce such reports from raw laparoscopic footage. The pipeline first labels instruments and anatomical targets in each frame with a Vision Transformer, then converts those labels into frame-level captions by fusing visual and textual embeddings, then uses a ViViT-based temporal encoder to generate clip-level summaries, and finally feeds those summaries to GPT-4 to compose a structured operative report. The authors report 96% precision in tool detection and BERT scores of about 0.74–0.81 for the captioning stages, arguing that the modular design keeps each decision step inspectable and reduces hallucination. A sympathetic reading is that a hierarchy of vision, temporal modeling, and language generation is enough to translate continuous surgical video into coherent clinical text.","feed_headline":"Three-stage AI pipeline writes surgery reports from video","feed_subtitle":"Vision transformers detect tools, ViViT captions clips, GPT-4 composes the final operative note.","key_machinery":"The load-bearing mechanism is the hierarchical fusion of visual and textual features across three time scales, carried by two named architectures: ViT for single-frame patches and ViViT, a video vision transformer that extends spatial patches with a temporal dimension, for clips of 32 frames with 16-frame overlap. Frame captions are produced by a cross-attention decoder (T5) over concatenated ViT and DistilBERT embeddings; clip captions use ViViT's spatiotemporal tokens fused with the frame captions; and the final report is generated by GPT-4 from a prompt that instructs it to merge repeated activities and combine their durations. The second part of the mechanism is two-phase training, in which the frame captioner is fine-tuned on the object detector's predicted labels and the clip captioner on the frame captioner's outputs, so each stage learns to tolerate the noise of the stage before it.","core_discovery":"On the paper's own terms, the central discovery is that surgical video can be turned into a structured written report by a cascaded multimodal transformer, where each stage consumes the previous stage's output rather than raw video alone. Frame-level object detection identifies tools and targets; frame captions are generated by concatenating ViT visual embeddings with DistilBERT object-label embeddings and decoding with T5; clip captions come from a ViViT model that sees 32-frame clips plus the generated frame captions; and GPT-4 merges the clip captions into a single narrative with phase durations. The reported evidence is that this cascade reaches 96% precision in instrument detection, BLEU 0.73 for frame captions, BLEU 0.67 for clip captions, and improves further after fine-tuning the captioners on detector outputs instead of ground-truth labels, which the paper presents as a way to absorb error propagation.","pith_inferences":["The evaluation splits frames, not videos, so adjacent frames from the same procedure appear in both training and test sets; the reported 96% tool precision therefore likely measures near-duplicate frame recognition rather than generalization to a new patient's surgery, and a video-level split would be the decisive test.","The reference frame and clip captions are templated sentences built from the same action, target, and phase annotations the model receives as input, so BLEU and BERTScore here measure how faithfully the model reproduces the template, not whether the summary is clinically true or complete.","Because the final report is written by GPT-4 from captions, a surgeon would be verifying the narrative against the captions, not the raw video; the trust boundary of the system is therefore at the clip-caption stage."],"forward_implications":["If the reported precision and BERT scores hold, per-frame tool and target labeling at one frame per second is accurate enough to feed a captioning chain that produces readable phase-by-phase operative notes.","Fine-tuning each captioning stage on the previous stage's predicted outputs rather than gold labels closes most of the quality gap caused by error propagation.","The same three-stage object-to-frame-to-clip-to-report architecture should transfer to other surgical procedures once phase, action, and instrument annotations exist for training.","Because the final report is built from intermediate captions, it becomes possible to audit the report by inspecting the tool detections and clip captions, supporting explainability in clinical use."],"supporting_citations":[{"why":"Supplies the CholecT50 dataset, the 50 laparoscopic videos and frame-level instrument, action, target, and phase annotations that the whole pipeline is trained and evaluated on.","marker":"[9]"},{"why":"Provides the ViT-base-patch16-224-in21k model used as the frame-level vision encoder for object detection and captioning.","marker":"[20]"},{"why":"Provides the ViViT video vision transformer that models temporal dependencies across 32-frame clips.","marker":"[4]"},{"why":"Provides the T5 text-to-text architecture used to generate frame-level and clip-level captions.","marker":"[52]"},{"why":"Supplies GPT-4, the large language model that composes the final structured surgical report from clip captions.","marker":"[47]"},{"why":"SurgT is the baseline model for surgical action triplet recognition that the object detection results are compared against.","marker":"[31]"},{"why":"Defines BERTScore, the semantic similarity metric used to evaluate frame and clip caption quality.","marker":"[68]"},{"why":"Provides DistilBERT, the lightweight textual encoder that processes detected object labels for multimodal fusion.","marker":"[55]"}],"fun_headline_variants":["Surgical video to full report via AI cascade","Transformer trio writes operative notes from video","AI pipeline generates surgical reports from clips","Multimodal AI turns laparoscopy video into reports","Three-stage AI drafts surgical documentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported generalization assumes a random frame-level split treats frames from the same surgical video as independent samples, and that templated captions derived from the annotations are valid ground truth for judging summary quality; if either fails, the headline numbers overstate real-world performance.","fun_headline_variants_meta":{"raw":{"variants":["Surgical video to full report via AI cascade","Transformer trio writes operative notes from video","AI pipeline generates surgical reports from clips","Multimodal AI turns laparoscopy video into reports","Three-stage AI drafts surgical documentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1415,"prompt_tokens":979,"completion_tokens":436,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":371}},"tokens_in":595,"tokens_out":436,"duration_ms":4993,"temperature":1.0,"reasoning_tokens":371,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:39:30.788631+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any held-out surgical video, run the pipeline, and compare tool detection precision on clips that share no frames with training, and the BERT score of clip captions against independently written surgeon reports instead of the templated captions; a substantial drop in either number would show the reported performance depends on frame leakage and template-matching rather than genuine generalization.","supporting_citations":[{"cited_title":"Cholect50: A dataset for surgical video understanding, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the CholecT50 dataset, the 50 laparoscopic videos and frame-level instrument, action, target, and phase annotations that the whole pipeline is trained and evaluated on."},{"cited_title":"Vivit: A video vision transformer","cited_arxiv_id":null,"evidence_quote":"Provides the ViViT video vision transformer that models temporal dependencies across 32-frame clips."},{"cited_title":"Exploring the limits of transfer learning with a unified text-to-text transformer","cited_arxiv_id":null,"evidence_quote":"Provides the T5 text-to-text architecture used to generate frame-level and clip-level captions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SurgT is the baseline model for surgical action triplet recognition that the object detection results are compared against."},{"cited_title":"Weinberger, and Yoav Artzi","cited_arxiv_id":null,"evidence_quote":"Defines BERTScore, the semantic similarity metric used to evaluate frame and clip caption quality."},{"cited_title":"Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter","cited_arxiv_id":null,"evidence_quote":"Provides DistilBERT, the lightweight textual encoder that processes detected object labels for multimodal fusion."}],"review_version":1}