{"id":"0f121d76-3334-4c68-ad89-88c4937cd8d5","arxiv_id":"2505.07449","paper_version":7,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Ophora generates ophthalmic surgical videos from natural language instructions using a curated 160K-pair dataset and progressive tuning of a pretrained video diffusion model.","lead":"Researchers built Ophora, an AI model that turns typed surgical instructions into realistic videos of eye surgery, trained on 160,000 video clips gathered from narrated operation videos. It is meant to give surgical AI systems more training data without needing to collect and annotate more real patient videos.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central quantitative claims may be inflated by data leakage: Ophora-160K's train/test split is not specified as video-level, and no analysis shows its training sources are disjoint from OphNet test videos.","rationale":"I disagree with the reader's choice of weakest assumption. The reader focuses on the accuracy of Qwen2.5-72B-refined instructions in Section 2.1. While that is a plausible data-quality risk, it does not directly support or undermine the strongest quantitative claim in the paper: the downstream workflow-understanding improvement in Table 2, because the videos used for augmentation are generated from phase labels via instructions written by three ophthalmologists, not from the LLM-refined instructions. If the refined instructions were noisy, the generator might follow instructions less precisely, but the downstream experiment would be largely insulated. A more load-bearing concern is dataset disjointness. The paper's own description is ambiguous: 'The dataset was split into 80% and 20% for training and testing' (Sec. 3) does not say video-level, and no overlap analysis between Ophora-160K training data and OphNet is reported. Both datasets draw from publicly available ophthalmic surgical videos, so the possibility of shared source videos or surgical cases is substantial and unaddressed. If leakage exists, the Table 2 gains and even the Table 1 quality metrics are inflated, invalidating the core claims of realism and downstream utility. This is a concrete, checkable ambiguity in the manuscript rather than a disagreement with external consensus. The proposed test would settle it. The reader's additional points about missing SurgEN baseline, missing error bars, and OphCLIP circularity are valid but secondary; error bars would not fix a leakage confound, and the SurgEN baseline would only contextualize, not validate, the downstream result. Therefore I recommend keeping the CONDITIONAL verdict, but with the condition explicitly on demonstrating video-level disjointness between Ophora training sources and the OphNet test set, rather than on the LLM instruction quality.","tokens_in":8883,"tokens_out":7622,"duration_ms":73982,"concrete_test":"Obtain source video identifiers for every Ophora-160K clip and for every OphNet train/val/test video. (1) Verify that the Ophora-160K train/test split is at the source-video level; if it was clip-level, recompute Table 1 with a strict video-level split and report the change in FID/FVD/CS. (2) Run near-duplicate retrieval between all Ophora-160K training clips and all OphNet test videos using perceptual hashing or frame-level feature matching. If any overlap or near-duplicate is found, retrain the classifiers in Table 2 using Ophora-generated videos from a model trained only on source videos absent from OphNet test, and compare the accuracy gain. If the 37.92-to-42.24 gap largely disappears, the downstream claim is an artifact of data leakage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's strongest quantitative evidence is the downstream improvement in Table 2 (MViTv2 phase Top-1: 37.92% to 42.24% on OphNet test). This gain is only meaningful if the Ophora-generated training videos do not leak OphNet test-domain identity. Yet the paper never establishes disjointness: Ophora-160K is built from OphVL, which collects public ophthalmic surgery videos, and OphNet is also assembled from public surgical videos. If any Ophora-160K training clip shares a source video (or even the same surgical case) with the OphNet test set, the classifier may have effectively trained on test-domain appearance, making the reported improvement spurious. Similarly, Section 3 states 'The dataset was split into 80% and 20% for training and testing' without specifying whether the split is at the source-video level. Because Ophora-160K consists of short clips cut from longer surgeries, a clip-level split places adjacent clips from the same surgery into both train and test, inflating the FID/FVD/CLIPScore results in Table 1 by measuring near-duplicate temporal content rather than generalization. The reader's concern about LLM-refined instruction accuracy is real, but it does not directly threaten Table 2 because those downstream videos are generated from ophthalmologist-written instructions, not Qwen-refined ones. The unresolved overlap issue is more load-bearing for both parts of the central claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Ophora, a text-guided video generation model for ophthalmic surgery. The authors construct Ophora-160K, a dataset of over 160K video-instruction pairs, by curating OphVL narrative videos: they use Qwen2.5-72B to rewrite narrative captions into generation instructions, apply dynamics-based filtering, and remove low-resolution clips. The model is based on CogVideoX-2b and is trained in two stages: transfer pre-training on the full Ophora-160K, followed by privacy-preserving fine-tuning on Ophora-28K, which excludes clips flagged by Qwen2.5-VL-72B as containing subtitles or watermarks. The authors evaluate video quality with FID, FVD, and CLIPScore, collect ophthalmologist ratings, and test downstream value by using Ophora-generated videos as augmented training data for OphNet workflow understanding, reporting improved phase-level and operation-level Top-1/Top-5 accuracy.","tokens_in":9185,"tokens_out":3156,"duration_ms":29935,"significance":"If the results hold, the paper makes a useful contribution: a large curated ophthalmic surgical video-instruction dataset, a privacy-aware two-stage tuning approach for adapting a natural video T2V model to a medical domain, and evidence that synthetic surgical videos can improve downstream workflow understanding. The release of code and the dataset is a practical strength, as is the inclusion of ophthalmologist feedback. However, the central quantitative claims are weakened by the lack of source-level split guarantees, potential data overlap between Ophora-160K and OphNet, and the use of a circular text-video alignment metric. These issues leave the strongest claims (Table 1 and Table 2) not fully supported.","major_comments":[{"comment":"The downstream augmentation result is load-bearing, but the paper does not establish that the Ophora-160K training sources are disjoint from the OphNet test set. OphVL collects public ophthalmic surgery videos and OphNet is also assembled from public videos. If Ophora-generated videos used for augmentation were trained on clips from the same source videos or same surgical cases as the OphNet test split, the reported Top-1 accuracy gain from 37.92% to 42.24% could reflect memorization of test-domain appearance rather than genuine generalization. The paper must provide a source-level or surgery-level disjointness analysis, or explicitly justify why overlap is impossible.","section":"Section 3, Table 2 and Dataset description"},{"comment":"The sentence 'The dataset was split into 80% and 20% for training and testing' does not specify the granularity of the split. Since Ophora-160K consists of short clips cut from longer narrative videos, a clip-level split places temporally adjacent clips from the same surgery in both training and test partitions. This can inflate the FID, FVD, and CLIPScore numbers in Table 1 by measuring near-duplicate temporal content, and it can also leak training information into the test set. The split should be performed at the source-video or surgery level, or the authors should justify why clip-level splitting is sufficient.","section":"Section 3, Dataset split"},{"comment":"The CLIPScore is computed with OphCLIP, which is developed on OphVL narrative data from which Ophora-160K is derived. Since Ophora is trained on the same narrative distribution, the text-video alignment score may be biased in favor of Ophora by construction. This makes the CS column in Table 1 a weak form of evidence for instruction following. The authors should report scores with an independent CLIP variant or a CLIP model not fine-tuned on OphVL data, and should also present error bars or significance tests over multiple generated video samples.","section":"Section 3, Quantitative Analysis and equation for CLIPScore"},{"comment":"The LLM-based refinement uses only 10 hand-annotated examples to instruct Qwen2.5-72B on rewriting narrative captions into generation instructions, but no validation is provided that the rewritten instructions preserve procedure-critical details or maintain temporal alignment with video content. Since Ophora-160K is the foundation of the model's instruction-following capability, the paper should include a human evaluation of a random sample of refined instructions to confirm their accuracy, completeness, and clinical correctness.","section":"Section 2.1, Narrative Information Refining"},{"comment":"The paper does not compare against SurgEN [4], a text-guided diffusion model for surgical video generation that is cited in the introduction as related work. Given the paper's claim of being a text-guided ophthalmic surgical video generation model, a direct comparison with SurgEN on quality or downstream metrics is necessary to substantiate the stated advantage over existing text-guided surgical generation approaches.","section":"Section 3, Quantitative Analysis and References"}],"minor_comments":[{"comment":"The phrase 'privacy-preserved ophthalmic surgical video generation' is used in the abstract and introduction, but the privacy mechanism described later only removes subtitles and watermarks. Clarify the scope of the privacy guarantee, since removing these artifacts does not necessarily prevent re-identification from surgical content.","section":"Abstract and Section 1"},{"comment":"The abbreviation P2FT is used in the figure but is not defined in the figure caption or in the main text before its usage. Define it at first mention in Section 2.3.","section":"Figure 1 caption"},{"comment":"The row labeled 'OphVL [12]' is ambiguous: it appears to be a dataset setting, but the table caption describes model comparisons. Clarify what is being evaluated in each row and what the column 'OphVL [12]' indicates.","section":"Table 1"},{"comment":"The caption marks Endora and Bora with an asterisk indicating fine-tuning on Ophora-160K, but the main text states that Endora is an unconditional generation model. This makes the comparison somewhat less direct; please state the conditionality of each model explicitly in the caption or text.","section":"Figure 2 caption"},{"comment":"The paper states that 600 videos were generated based on Cataract-1K phase labels, but it does not specify how many videos were generated per instruction or whether the ophthalmologist-written instructions were verified before generation. Adding this detail would help reproducibility.","section":"Section 3, Ophthalmologist Feedback"},{"comment":"Several references are cited in an incomplete or non-uniform style (e.g., reference [10] and [12] have publisher information; others lack it). Please ensure all references follow a consistent journal format.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main technical concern is the unresolved possibility of data overlap between the Ophora training set and the OphNet test set, which directly affects the credibility of the downstream accuracy improvements reported in Table 2. I would ask the editor to require the authors to provide a source-level disjointness analysis or a re-run of the downstream experiment with a suitably restricted training set before considering acceptance. The CLIPScore circularity is also a concern but is more tractable; reporting scores with an external CLIP model would address most of it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before citing its numbers: Ophora is a real engineering effort—a 162K clip-instruction ophthalmic surgery dataset and a CogVideoX-2b model fine-tuned in two stages—but the evaluation as written does not rule out leakage between the synthetic-augmentation training data and the OphNet test set. That makes the headline +4.3 point Top-1 gain (37.92→42.24) plausible but not yet trustworthy.\n\nWhat is genuinely new: Ophora-160K is, as far as I know, the first large-scale text-video dataset for ophthalmic surgery. The curation pipeline—LLM-based caption rewriting with a 10-example in-context prompt, dynamics-based keyframe filtering, resolution floor—is sensible. The two-stage training (transfer pre-training on the full set, then privacy-preserving fine-tuning on the 28K clean subset) is a reasonable design, and releasing code is good practice. The ophthalmologist feedback is a useful addition, and FID/FVD are standard.\n\nThe soft spots are in the evaluation, not the construction. Section 3 says the dataset was split 80/20 but never says the split is video-level. Ophora-160K has 16.5 clips per source video on average, so a clip-level split would put near-duplicates in both train and test, inflating Table 1. More importantly, Table 2's downstream gain is only interpretable if Ophora's training sources are disjoint from OphNet's test videos. Both are assembled from public surgical videos; the paper shows no such disjointness analysis. If the generated augmentation videos carry test-domain appearance, the accuracy jump could be memorization rather than useful synthesis. That is load-bearing, and the authors should be asked to address it.\n\nAlso missing: SurgEN, the most relevant text-guided surgical video baseline, is cited but never compared. The CLIPScore uses OphCLIP, from the same group and data lineage as the training captions, which is at least a suspicious choice for an alignment metric. There are no error bars or significance tests. The privacy claim is overstated: filtering subtitles and watermarks does not guarantee that the model hasn't memorized patient-specific visual features. The unvalidated LLM-refined instructions are a real concern for the dataset's semantic correctness, though it doesn't directly threaten Table 2, where prompts were written by ophthalmologists.\n\nOverall, the core idea is sound and the dataset is a useful community asset. But the paper's strongest quantitative claim is not yet supported. It should go to peer review with a request for video-level split details, a disjointness analysis against OphNet, a SurgEN comparison, and variance estimates.\n\nFor whom: surgical AI and medical video generation researchers. Worth a reading-group slot on evaluation pitfalls in generative medical video.\n\nRecommendation: engage, but push on the leakage issue.","headline":"A genuinely useful dataset and a plausible tuning recipe, but the headline numbers are not airtight until the authors rule out video-level leakage between Ophora and OphNet.","tokens_in":9768,"tokens_out":4634,"would_cite":false,"duration_ms":41570,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ophora generates realistic ophthalmic surgical videos from natural-language instructions, and using them as training data lifts phase-recognition accuracy on the OphNet benchmark from 37.92% to 42.24%.","keywords":["ophthalmic surgery","text-guided video generation","video diffusion model","data curation","instruction tuning","privacy preservation","surgical workflow understanding","data augmentation"],"falsifier":"Take 100 held-out instruction prompts from the Ophora test split, generate videos with Ophora, and have two independent ophthalmologists rate each video on a fine-grained checklist: correct instrument used, correct tissue/anatomy affected, and correct order of sub-actions. If the mean score on any of those categories falls below 2 on the paper's own 0-3 realism scale, the claim that Ophora reliably follows surgeon instructions would be contradicted.","tokens_in":8660,"feed_emoji":"👁️","tokens_out":11205,"duration_ms":88089,"temperature":0.7,"pith_summary":"The paper presents Ophora, a text-guided video generation model that synthesizes ophthalmic surgical videos from natural-language instructions. To train it, the authors built Ophora-160K, a dataset of over 160K video-instruction pairs, by refining captions from narrated surgical videos with a large language model and filtering clips on visual dynamics and resolution. They then progressively tuned a pre-trained video diffusion model, first on the full dataset and then on a privacy-cleaned subset, to transfer natural-video knowledge to the surgical domain. The paper's central claim is that the resulting videos are realistic and instruction-following, as judged by automated metrics and ophthalmologist ratings, and that they are useful as training data: adding them to a real surgical benchmark raises phase-level Top-1 accuracy from 37.92% to 42.24% for an MViTv2 classifier on the OphNet test set.","feed_headline":"Synthetic eye-surgery videos lift surgical AI accuracy by 4.3 points","feed_subtitle":"A text-guided model trained on 160K curated clips produces realistic ophthalmic videos and improves phase recognition.","key_machinery":"The argument turns on two coordinated mechanisms. The Comprehensive Data Curation pipeline turns narrative ophthalmic videos into a large instruction-following dataset: Qwen2.5-72B, prompted with ten hand-written examples, rewrites verbose narrative captions into concise generation instructions; PySceneDetect keyframe counts are used to discard clips with extreme dynamics (fewer than 2 or more than 100 keyframes); and clips below 720×480 are removed. The Progressive Video-Instruction Tuning scheme then adapts CogVideoX-2b, a latent diffusion model pre-trained on natural video-text pairs, to surgery: transfer pre-training trains only the denoising network on all of Ophora-160K while the T5 text encoder and 3D VAE stay frozen, and privacy-preserving fine-tuning continues on Ophora-28K, a subset cleared by Qwen2.5-VL-72B of frames carrying subtitles or watermarks. The training objective is the standard diffusion denoising loss of Eq. (1).","core_discovery":"Ophora, built from the CogVideoX-2b latent diffusion backbone, can generate realistic, instruction-following ophthalmic surgical videos and does so without reproducing sensitive overlays such as subtitles or watermarks. The authors establish this by constructing the Ophora-160K dataset from OphVL's narrative videos through LLM-based instruction refinement and keyframe-dynamics filtering, then applying two-stage tuning: transfer pre-training on the full dataset followed by privacy-preserving fine-tuning on a filtered 28K subset. They report that Ophora outperforms the Endora and Bora baselines on FID, FVD, and OphCLIP-based CLIPScore, and that three ophthalmologists rate its generations as realistic across seven criteria covering phase, construction, and action. Finally, they show a downstream benefit: using Ophora-generated videos as additional training data improves phase and operation recognition on the OphNet benchmark across both SlowFast and MViTv2 classifiers.","pith_inferences":["A direct testable extension is applying the same curation-plus-tuning recipe to other surgical video archives; the pipeline's components are not ophthalmic-specific, though the paper does not claim this.","The downstream accuracy gain may reflect increased training set size rather than the realism of Ophora's videos; holding the number of augmented videos constant while varying their source (e.g., real vs. Ophora vs. an untuned generator) would isolate the mechanism.","Given the 10-example instruction rewrite stage, regenerating instructions with more expert-annotated examples or adding a surgical-knowledge filter would likely change both FVD/CLIPScore and downstream gains, a comparison the paper does not run.","The model's clips average 5.54 seconds, so whether multi-phase, longer-duration generation preserves cross-phase instrument and tissue dynamics is an open question that the paper's future-work statement acknowledges."],"forward_implications":["If the claims hold, surgical AI training can use Ophora-generated videos to augment scarce real recordings, with a measured phase-level Top-1 gain from 37.92% to 42.24% on the OphNet test set for MViTv2.","Because the final fine-tuning step removes subtitles and watermarks, the pipeline produces shareable video data that avoids the privacy obstacles attached to real ophthalmic recordings.","The instruction-following capability supports on-demand generation of specific procedural steps, which the authors position as a resource for training and rehearsal.","The authors state the same approach can be extended to other surgery types and longer-duration videos, a direction they list as future work."],"supporting_citations":[{"why":"CogVideoX-2b is the pre-trained text-to-video latent diffusion model whose denoising network is progressively tuned; it supplies the spatial-temporal prior.","marker":"[27]"},{"why":"OphVL provides the narrative ophthalmic video-caption pairs that become Ophora-160K, and OphCLIP is used to compute the CLIPScore metric.","marker":"[12]"},{"why":"Qwen2.5-72B is the LLM that rewrites narrative captions into generation instructions using ten hand-annotated examples.","marker":"[26]"},{"why":"Qwen2.5-VL-72B detects subtitles, watermarks, and other sensitive content to build the privacy-preserved Ophora-28K subset.","marker":"[1]"},{"why":"OphNet is the benchmark used to evaluate downstream surgical workflow understanding with augmented training data.","marker":"[11]"},{"why":"MViTv2 is the classifier whose phase-level Top-1 accuracy rises from 37.92% to 42.24% when Ophora-generated videos are added to training.","marker":"[18]"},{"why":"Bora is the biomedical text-guided video generation baseline that Ophora outperforms on FID, FVD, and CLIPScore and in downstream augmentation.","marker":"[22]"},{"why":"Endora is the unconditional surgical video generation baseline included in the quantitative quality comparison.","marker":"[17]"}],"fun_headline_variants":["Ophora: text prompts become realistic eye-surgery videos","Synthetic surgery clips from text improve ophthalmic AI","Privacy-safe AI generates eye surgery videos to train models","Text-guided video gen boosts surgical phase recognition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the LLM-rewritten instructions are medically accurate, aligned with each clip, and free of clinically meaningful errors, since the rewriting was taught with only ten hand-annotated examples and was never validated against a ground-truth surgical description.","fun_headline_variants_meta":{"raw":{"variants":["Ophora: text prompts become realistic eye-surgery videos","Synthetic surgery clips from text improve ophthalmic AI","Privacy-safe AI generates eye surgery videos to train models","Text-guided video gen boosts surgical phase recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000492,"raw_usage":{"total_tokens":2443,"prompt_tokens":992,"completion_tokens":1451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":1387}},"tokens_in":608,"tokens_out":1451,"duration_ms":10454,"temperature":1.0,"reasoning_tokens":1387,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:16:05.315737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 100 held-out instruction prompts from the Ophora test split, generate videos with Ophora, and have two independent ophthalmologists rate each video on a fine-grained checklist: correct instrument used, correct tissue/anatomy affected, and correct order of sub-actions. If the mean score on any of those categories falls below 2 on the paper's own 0-3 realism scale, the claim that Ophora reliably follows surgeon instructions would be contradicted.","supporting_citations":[{"cited_title":"In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G","cited_arxiv_id":null,"evidence_quote":"OphNet is the benchmark used to evaluate downstream surgical workflow understanding with augmented training data."},{"cited_title":"In: Linguraru, M.G., Dou, Q., Feragen, A., Giannarou, S., Glocker, B., Lekadir, K., Schnabel, J.A","cited_arxiv_id":null,"evidence_quote":"Endora is the unconditional surgical video generation baseline included in the quantitative quality comparison."}],"review_version":1}