{"id":"ee1396c5-01d8-41ef-afca-190d5c4d8df4","arxiv_id":"2412.17346","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"FFA Sora generates simulated fundus fluorescein angiography videos from text reports using an Open-Sora-based diffusion transformer, with moderate objective metrics and weak evidence for the privacy claim.","lead":"This paper builds FFA Sora, a text-to-video model that turns written eye-exam reports into simulated fluorescein angiography videos of the retina. The goal is to let trainees practice reading these videos and to share synthetic medical data without exposing real patient information.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No patient/eye-level split is described for 3,625 videos sharing only 1,814 reports, so random video-level splitting can leak the same eye into train and test, inflating quality metrics and invalidating the privacy claim; the central claim depends on this being fixed.","rationale":"The reader's weakest_assumption correctly notes the ambiguity of report-video pairing and the absence of a patient/eye-level split. I agree that this is a central weakness, but I would isolate the missing split as the single most load-bearing issue because it can invalidate both headline claims: the quality metrics may be inflated by memorization, and the privacy claim is directly contradicted if the model can reproduce a patient's vasculature. The report-video mismatch is important for the text-conditioning argument, but the split also affects the integrity of every objective and subjective evaluation. This is not an ad hominem; it is a standard data-leakage concern that can be settled by a controlled re-run with proper metadata. The reader's CONDITIONAL verdict remains appropriate: the paper has a plausible pipeline and transparently reports dataset sizes and metrics, but the missing split is a blocking issue that a revision could address with an eye-level split, confidence intervals, and a membership-inference analysis. I recommend no change to the reader's verdict; the conditional acceptance stands until this test is performed.","tokens_in":9523,"tokens_out":8449,"duration_ms":84813,"concrete_test":"Ask the authors to identify the mapping from each of the 3,625 videos to a unique patient/eye (using the 1,814 reports or DICOM metadata), then re-run all training and evaluation with a strict eye-level split: no eye appears in both training and test. Recompute FVD, LPIPS, human scores, and Recall@K. If the quality metrics worsen significantly or the Recall@K increases, the original results were contaminated by train/test leakage; if they remain stable, the concern is resolved. As a complementary check, run a membership-inference test: measure nearest-neighbor distance from each generated video to training videos of the same report versus held-out videos; high proximity to training identities would refute the privacy claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing flaw is the absence of any patient/eye-level split. The paper reports 3,625 videos paired with 1,814 reports, and states only that the dataset was randomly divided 80/10/10 at the video level. Because FFA videos from the same eye/patient naturally share a report (1814 reports for 3625 videos), random video-level splitting places videos from the same source eye in both training and test folds with high probability. This has two consequences for the central claim. First, the quality metrics (FVD=329.78, human score 1.57) may reflect memorization of individual eyes rather than generalization of text-to-video synthesis: the DiT can copy vascular patterns it saw during training, inflating similarity to the ground-truth test videos. Second, the privacy claim, which is the paper's main motivation, is contradicted by the possibility of memorization: if the model can reproduce a patient's retinal vasculature, generated videos are not de-identified content, and the low Recall@K (mean 0.073) is not evidence of privacy because recall is computed against the test set and a weak feature extractor; it cannot detect copying of training identities. Without an eye/patient-level split or an analysis of the report-to-video mapping, neither 'accurately simulates disease features' nor 'strong privacy-preserving performance' is established. The report-sharing (1,814 reports for 3,625 videos) also means the text prompt may describe the whole examination while a given video covers a phase subset, weakening the text-conditioning premise and the BERTScore evaluation (0.35). Both issues are resolved by a proper split and a per-report video coverage analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents FFA Sora, a text-to-video diffusion transformer for generating 21-frame fundus fluorescein angiography (FFA) videos from textual clinical reports. The model combines a Wavelet-Flow Variational Autoencoder (WF-VAE) with a Diffusion Transformer (DiT) adapted from the Open-Sora Plan. The authors train on 3,625 FFA videos paired with 1,814 reports, randomly split 80/10/10 at the video level, and report FVD=329.78, LPIPS=0.48, VQAScore=0.61, BERTScore=0.35, average Recall@K=0.073, and a human rating of 1.57 (where 1 is best). The central claims are that the model accurately simulates disease features from text and preserves patient privacy, enabling synthetic FFA sharing for education and model training.","tokens_in":9775,"tokens_out":6614,"duration_ms":60247,"significance":"The application is novel and timely: text-to-video generation for FFA has clear educational and data-sharing value, and using a DiT for continuous FFA video generation is a reasonable extension of existing open-source video-diffusion methods. The authors should be credited for using established video-quality metrics, for making the base framework code available, and for attempting a domain-specific text-alignment evaluation. However, the reported evidence does not currently support the headline claims: the video-level random split with shared reports creates a leakage risk, the objective metrics lack baselines and uncertainty estimates, and the retrieval-based privacy evaluation is not a meaningful re-identification test. These issues are correctable with re-splitting and additional experiments, so I view this as a major revision rather than a rejection.","major_comments":[{"comment":"The dataset is stated to contain 3,625 FFA videos paired with only 1,814 reports, and the split is described as \"randomly selected\" 80/10/10 with no patient-level or eye-level stratification. Because multiple videos from the same eye or patient share a single report, a random video-level split places videos from the same source into both training and test sets with high probability. This identity leakage can inflate FVD, LPIPS, VQAScore, and the human score through memorization of individual vascular patterns, and it invalidates the retrieval-based privacy claim because the test set is not disjoint from training identities. The authors should re-split at the patient or eye level and report the number of unique patients or eyes in each fold, or otherwise demonstrate that the report-to-video mapping does not cross the split.","section":"Methods (Dataset) and Results (first paragraph)"},{"comment":"FVD=329.78, LPIPS=0.48, VQAScore=0.61, and BERTScore=0.35 are single point estimates with no confidence intervals, no multiple-seed variation, and no baseline values. The claim that these numbers indicate \"excellent quality\" and \"satisfactory performance\" requires comparison with, at minimum, the scores of real videos against themselves, a text-unconditioned or randomly conditioned generator, and a previously published video-generation model. Without such baselines, the absolute values are uninterpretable.","section":"Results, Table 1A"},{"comment":"The domain-specific alignment evaluation uses FFA-GPT (ref. 15) to translate generated videos back into text and then computes BERTScore against the input prompt. FFA-GPT is a model developed by the same group on the same dataset, so this is an internal text-video-text roundtrip rather than an external clinical benchmark. Moreover, BERTScore=0.35 is low in absolute terms, and no reference BERTScore is given for real videos under the same pipeline. The evaluation should be repeated with an independent report-generation model or with masked clinical readers, and reference scores should be reported.","section":"Objective Evaluation and Results (Figure 3)"},{"comment":"The privacy claim rests on low Recall@K in an image-to-image retrieval task against ground-truth test videos. Low recall under a single feature extractor only shows that generated videos are not top-K nearest neighbors of the corresponding test videos in that embedding space; it does not test for training-set memorization, membership inference, or re-identification of patients whose videos appeared in training. Given the video-level split leakage described above, the reported Recall@K values do not establish \"strong privacy-preserving performance.\" The authors should add a direct memorization test, such as nearest-neighbor search of generated videos against the full training set, and a membership-inference experiment.","section":"Results, Table 1B (privacy evaluation)"},{"comment":"The human evaluation was performed by three authors (X.W., X.Y., Y.F.) on 50 generated videos, without blinding to the model outputs or to the study hypothesis, and without comparison against a baseline generator or against real videos. The resulting average score of 1.570 cannot support the claim of \"satisfactory visual quality\" as an unbiased clinical assessment. Independent masked raters, a pre-registered rating protocol, and inter-rater reliability statistics are needed.","section":"Human Assessment"}],"minor_comments":[{"comment":"The interpolation formula is written as F_interpolated = (1−t)×F_i − t×F_{i+1}, which contains a minus sign and would not produce a convex combination. This should be a plus sign: (1−t)×F_i + t×F_{i+1}.","section":"Methods (frame interpolation)"},{"comment":"The Code availability section links only to the Open-Sora Plan repository, not to the FFA Sora implementation, preprocessing scripts, or model weights. Please provide the actual project code to make the results reproducible.","section":"Code availability"},{"comment":"The text reports an average Recall of 0.073, while Table 1B lists 0.07; please make these consistent.","section":"Results and Table 1B"},{"comment":"The frame-preselection rule (\"selected 21 frames in reverse chronological order\") means the standardized videos emphasize venous and late phases; please clarify whether this temporal ordering matches the FFA examination protocol and how it affects the model's ability to generate the arterial phase.","section":"Methods (Dataset)"},{"comment":"The second limitation in the Limitations section states that \"our evaluation approach may not fully capture the broader generative quality.\" This is an important caveat that should be reflected in the abstract and Results claims, not only in the limitation paragraph.","section":"Limitations"},{"comment":"There is a typo \"angioagraphy\" in the Discussion; please proofread the manuscript.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern is valid and, on reading the full text, is the central issue: the random video-level split with 1,814 reports for 3,625 videos creates a real leakage risk for both the quality and privacy claims. I also agree with the circularity concern regarding FFA-GPT-based BERTScore. The paper is not ready for publication in its current form, but the flaws are fixable with re-splitting and additional evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first text-to-video model for fundus fluorescein angiography, and the authors show that a fine-tuned Open-Sora stack can generate recognizable FFA-like clips from report text. Second, the evidence for the two headline claims—'accurately simulates disease features' and 'strong privacy-preserving performance'—does not support those claims as written.\n\nWhat is genuinely new: the domain application. Adapting WF-VAE with CausalConv3D and a DiT to medical angiography video is a sensible use of existing components, and the paper is transparent about its architecture. The dataset, 3,625 real FFA videos with paired reports, is a legitimate clinical resource, and the filtering and frame-standardization steps are clearly described. That part is a solid engineering effort.\n\nThe soft spots are serious. The load-bearing problem is the data split. The paper says videos were randomly divided 80/10/10, but 3,625 videos are paired with only 1,814 reports, so multiple videos come from the same eye. Without a patient- or eye-level split, the same eye is almost certainly represented in both training and test folds. That can inflate every quality metric through memorization of individual vascular patterns, and it directly undermines the privacy claim—if the model can reproduce a patient's retinal vasculature, it is not de-identifying anything. The low Recall@K is not a privacy guarantee; it is a retrieval result against the test set with a weak feature extractor. It says nothing about whether training identities can be copied.\n\nBeyond that, the objective metrics are point estimates with no confidence intervals, no baselines from other generators, and no real-video self-comparison. BERTScore at 0.35 is low, and the text-video-text evaluation uses FFA-GPT, from the same group and the same dataset, so it is an internal check, not an external benchmark. The human raters are the authors, which is a conflict, and there is no inter-rater agreement. The report-to-video mapping is also ambiguous: a report may describe the whole examination, while a given 21-frame video covers one phase, which weakens the text-conditioning premise.\n\nThe limitations section is honest about needing external validation, but the abstract and results overstate what is shown.\n\nWho should read it: anyone working on generative models for ophthalmology or on privacy-preserving synthetic medical data. It is a useful existence proof, not a validated clinical tool.\n\nMy recommendation: yes, send it to peer review—the topic is worth referee time and the main flaws are identifiable and fixable with a patient-level split, baselines, error bars, and a formal re-identification analysis. But as it stands, the central claims are not established.","headline":"First text-to-video FFA simulator, but the split and evaluation don't support the accuracy or privacy claims.","tokens_in":10457,"tokens_out":2925,"would_cite":false,"duration_ms":28135,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FFA Sora turns text reports into 21-frame synthetic fluorescein angiography videos and reports low patient re-identification.","keywords":["fundus fluorescein angiography","text-to-video generation","diffusion transformer","wavelet flow variational autoencoder","privacy-preserving data sharing","medical education","retinal vascular disease","synthetic video evaluation"],"falsifier":"Generate clips from prompts that name one lesion in one retinal quadrant and a specific phase, then have masked ophthalmologists or an automated detector localize that lesion in the generated frames; if localization accuracy is no better than chance, the claim that FFA Sora accurately simulates disease features from text fails.","tokens_in":9227,"feed_emoji":"🎥","tokens_out":9686,"duration_ms":84143,"temperature":0.7,"pith_summary":"FFA Sora is a text-to-video model that takes a written fundus fluorescein angiography (FFA) report and turns it into a 21-frame synthetic FFA video showing retinal vascular structures and lesions described in the text. The paper's central claim is that the generated videos accurately simulate disease features such as dye leakage, microaneurysms, and non-perfusion areas, while producing low image-retrieval recall (average Recall@K = 0.073) that the authors interpret as strong privacy preservation against patient re-identification. The evidence comes from a retrospective dataset of 3,625 videos paired with 1,814 reports, objective quality metrics (FVD = 329.78, LPIPS = 0.48, VQAScore = 0.61), a BERTScore of 0.35 between reports regenerated from generated videos and the original prompts, and a human rating of 1.570 on a scale where 1 is best. If the claim holds, hospitals could share synthetic FFA sequences for medical education and for training diagnostic AI without releasing patient-identifiable video.","feed_headline":"Text-to-video model turns FFA reports into synthetic angiography films","feed_subtitle":"Generated 21-frame videos track retinal dye leakage and lesions while scoring low on patient re-identification.","key_machinery":"The carrying mechanism is a two-stage latent video generator adapted from an open-source text-to-video implementation. The Wavelet-Flow Variational Autoencoder (WF-VAE) compresses FFA clips into latent tensors, using wavelet decomposition to route low-frequency image energy through a Main Energy Flow Pathway while converting convolutional layers to CausalConv3D so the model sees temporal structure. A Diffusion Transformer (DiT) then performs iterative denoising in that latent space, with self-attention for spatial and temporal coherence, cross-attention to text embeddings from a pretrained text encoder, and gating mechanisms that refine feature fusion. The decoder reconstructs 21-frame, 512-by-512 videos, and the pipeline includes a frame-standardization step that uses reverse-chronological selection or linear interpolation to fix every input to 21 frames.","core_discovery":"The central discovery, on the paper's own terms, is that a diffusion transformer conditioned on text can synthesize FFA video sequences that carry the same clinically relevant features as real examinations—leakage, neovascularization, microaneurysms, capillary non-perfusion, and macular edema—across common retinal and choroidal diseases, and that the synthesized videos are hard to match back to their source videos in a retrieval test. The paper demonstrates this by building FFA Sora, applying it to a held-out test set, and reporting that the generated videos score well on standard generative metrics, achieve an acceptable text-alignment score, and receive an average visual quality rating of 1.570, close to the best possible score of 1.","pith_inferences":["A stronger fidelity test than BERTScore would be lesion localization: prompting for 'microaneurysms in the superior arcade' and asking a masked grader to draw the lesion should place it at the specified location; the paper does not report location-level correspondence.","The low Recall@K could also arise from generic, template-like videos that share little with any specific patient, so the privacy result is best read as 'no easy re-identification in this retrieval setup' rather than a guarantee against a trained re-identification adversary.","Because videos longer than 21 frames are subsampled in reverse chronological order, the generated sequence's temporal ordering need not follow the real fluorescein bolus phases; a direct check is whether the venous phase reliably precedes the late phase in generated clips.","If the model truly learns text-conditioned disease dynamics, the same architecture should work for other angiographic modalities, such as indocyanine green angiography, with minimal changes to the text encoder and prompt vocabulary."],"forward_implications":["Text prompts describing diabetic retinopathy, vein occlusion, and choroidal diseases can be turned into synthetic FFA videos suitable for teaching trainees to recognize dynamic dye behavior.","The low average Recall@K of 0.073 implies that synthetic FFA videos could be shared across institutions with a much smaller risk of re-identifying the patient behind the original video.","Because a report-generation model can regenerate a textual report from a generated video with BERTScore 0.35 against the prompt, the model offers a way to create large, labeled, text-video training pairs for downstream diagnostic AI.","The same DiT-based architecture can generate FFA videos for less common conditions such as uveitis and retinitis pigmentosa, broadening the range of teachable cases beyond what a single hospital can collect."],"supporting_citations":[{"why":"Supplies the FFA dataset and the automated report generator used to turn generated videos back into text for BERTScore evaluation.","marker":"[15]"},{"why":"Provides the open-source text-to-video generation codebase that FFA Sora adapts for the medical domain.","marker":"[16]"},{"why":"Defines the wavelet-driven flow video autoencoder with CausalConv3D that compresses FFA clips into latent space.","marker":"[17]"},{"why":"Introduces the transformer-based diffusion backbone with cross-attention used for text-conditioned video denoising.","marker":"[18]"},{"why":"Defines the Fréchet Video Distance metric used to compare generated and real video distributions.","marker":"[19]"},{"why":"Defines the perceptual image similarity metric used to measure frame-level fidelity.","marker":"[20]"},{"why":"Provides the text-to-visual alignment score used to evaluate prompt fidelity.","marker":"[21]"},{"why":"Defines the BERT-based text similarity score used to measure alignment between regenerated and original reports.","marker":"[22]"},{"why":"Supplies the image-retrieval Recall@K protocol used to measure re-identification risk and the human grading methodology.","marker":"[23]"},{"why":"Is the prior cross-modal FFA video generation work whose evaluation approach the human assessment follows.","marker":"[24]"}],"fun_headline_variants":["FFA Sora turns text reports into synthetic angiography videos","Synthetic FFA videos mimic dye leakage for medical education","Text prompts generate realistic retinal angiography videos","Diffusion transformer creates FFA videos from clinical reports","AI-simulated angiography preserves privacy while teaching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each text report is a specific and accurate description of its paired FFA video, even though 3,625 videos share only 1,814 reports and no patient-level or eye-level split is reported, so the text-to-video alignment and its evaluation rest on an ambiguous pairing.","fun_headline_variants_meta":{"raw":{"variants":["FFA Sora turns text reports into synthetic angiography videos","Synthetic FFA videos mimic dye leakage for medical education","Text prompts generate realistic retinal angiography videos","Diffusion transformer creates FFA videos from clinical reports","AI-simulated angiography preserves privacy while teaching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000501,"raw_usage":{"total_tokens":2431,"prompt_tokens":909,"completion_tokens":1522,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":1448}},"tokens_in":525,"tokens_out":1522,"duration_ms":12087,"temperature":1.0,"reasoning_tokens":1448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:33:48.307210+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate clips from prompts that name one lesion in one retinal quadrant and a specific phase, then have masked ophthalmologists or an automated detector localize that lesion in the generated frames; if localization accuracy is no better than chance, the claim that FFA Sora accurately simulates disease features from text fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the wavelet-driven flow video autoencoder with CausalConv3D that compresses FFA clips into latent space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the transformer-based diffusion backbone with cross-attention used for text-conditioned video denoising."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Fréchet Video Distance metric used to compare generated and real video distributions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the perceptual image similarity metric used to measure frame-level fidelity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the text-to-visual alignment score used to evaluate prompt fidelity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the BERT-based text similarity score used to measure alignment between regenerated and original reports."},{"cited_title":"Fundus to Fluorescein Angiography Video Generation as a Retinal Generative Foundation Model","cited_arxiv_id":"2410.13242","evidence_quote":"Is the prior cross-modal FFA video generation work whose evaluation approach the human assessment follows."}],"review_version":1}