{"id":"2b7f03e3-4318-4aba-a328-bfa5ca866ca7","arxiv_id":"2501.04325","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"IVEDiff performs image-guided video editing by inflating an image editing model with temporal motion modules and fine-tuning them with masked motion modeling.","lead":"This paper introduces IVEDiff, a diffusion model that edits a specific object in a video by referencing a single image and a mask. It adds motion modules plus an optical-flow-guided warp network, and a masked-motion fine-tuning strategy, to keep edited frames temporally consistent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal-consistency metrics use the same source optical flow that MotRefNet is designed to follow, so the reported smoothness gains may be partly circular; the paper lacks an independent consistency measure.","rationale":"The reader's weakest assumption flags that source optical flow may be an invalid propagation prior when the masked region is replaced by semantically unrelated content. I agree that this is a real gap, but I see a sharper and more directly testable issue: the temporal-consistency metrics themselves are computed with the same source flow that MotRefNet is trained to follow, so the reported smoothness advantage is partly circular. This does not mean the method is wrong; the ablation in Table 2 shows that MMM alone (Exp1) accounts for most of the Warp Error reduction, and MMM does not depend on source flow. However, the paper's headline quantitative evidence for temporal smoothness rests on a metric aligned with the model's inductive bias. An independent flow estimate on the edited output would settle whether the smoothness is real or metric-driven. The reader's concern about different object motion is related but distinct; it asks whether the prior is valid, while my concern asks whether the evidence would survive without sharing that prior. Both are addressable, and neither, if confirmed, necessarily refutes the method; they would require the authors to qualify the robustness claim and add an independent evaluation. The current CONDITIONAL verdict remains appropriate, so I recommend no change.","tokens_in":15106,"tokens_out":7536,"duration_ms":78955,"concrete_test":"Using the same 236 triplets and the generated edited videos, recompute Warp Error and Temporal Consistency with GMFlow optical flow estimated on the edited videos themselves (warp edited frame t to t+1) rather than on the source frames. If IVEDiff's advantage over AnimateDiffV3* shrinks or reverses, the temporal-smoothness claim is substantially metric-driven and the central claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is not that MotRefNet uses source optical flow as a design choice, but that the paper's evidence for temporal smoothness is computed with the same flow. In Section 4.2, MotRefNet predicts a 2-channel offset and warps the previous edited-frame latent according to GMFlow flow extracted from the source frames (Eqs. 3-6). In Section 5.1, Warp Error and Temporal Consistency are computed by warping edited frames with flow obtained from the source video, following references [24] and [52]. Hence a method that reproduces the source motion is rewarded by the metric, even if the edited object's appearance is not faithfully propagated. For object modification (Appendix A.1), the reference is semantically unrelated, the depth map is zeroed, and the mask is a coarse rectangle; the source flow is the only motion prior available to both the model and the metric. The paper contains no experiment in which the edited object is expected to move differently from the source object, and no independent temporal-consistency measure (e.g., flow estimated on the edited video). Consequently the central claim that IVEDiff is able to generate temporally smooth edited videos while robustly dealing with various editing objects is not yet established: the reported smoothness may reflect adherence to source motion rather than genuine visual coherence of the edited content. This is an evidence gap, not a demonstrated failure, and it is addressable by an independent evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IVEDiff, a diffusion model for image-guided video editing. Given a source video, per-frame masks, and a reference image, the model edits masked regions by injecting features from a reference UNet and uses a learnable MotRefNet to warp previous-frame latents along source optical flow; an MMM fine-tuning strategy randomly masks spatial regions of training clips while keeping the first frame as reference. The authors construct an IVE-Benchmark of 236 video-mask-reference triplets covering object modification and texture transfer, and compare against per-frame MimicBrush editing, AnimateDiffV3, and a fine-tuned variant on Warp Error, Temporal Consistency, FID, and CLIP Score. The paper claims to be the first image-guided video editing diffusion model and to achieve temporally smooth, high-quality edits.","tokens_in":15442,"tokens_out":6334,"duration_ms":59522,"significance":"If the reported results hold, IVEDiff is a practical contribution: it extends reference-based image editing to video with local control, introduces a sensible MMM objective that aligns fine-tuning with inference, and provides a reusable benchmark for an under-studied task. The ablation structure (Table 2) and consistent direction of improvements across metrics are strengths. However, the central claim of temporal smoothness rests on metrics that reuse the same source optical flow that guides the model, and the benchmark is self-constructed with hyperparameters selected on it; independent verification is therefore needed before the claim is established. The paper does not release code or data, limiting reproducibility.","major_comments":[{"comment":"The temporal-consistency evaluation is partly circular. In §4.2 (Eqs. 3-6), MotRefNet is trained to warp previous edited latents using GMFlow optical flow computed from the source frames. In §5.1, 'Warp Error' and 'Temporal Consistency' are computed by warping edited frames with optical flow obtained from the source video, following references [24,52]. Thus a model that reproduces source motion is rewarded by these metrics even if the edited object's appearance is not faithfully propagated. The paper provides no metric computed on the edited video itself (e.g., flow estimated between edited frames) and no experiment where the edited object moves differently from the source object. This is a load-bearing evidence gap for the abstract and conclusion claim of 'temporally smooth' editing, and it should be addressed with an independent evaluation before the central claim is accepted.","section":"§4.2 and §5.1, Eqs. (3)-(6), Table 1"},{"comment":"The key MMM hyperparameters (mask ratio in Tables 3 and 6, stride in Table 5, mask strategy in Table 4) are selected through ablations on the same IVE-Benchmark that is used to report final numbers. No held-out validation split, cross-validation, or error bars are provided. Because the final comparison in Table 1 and the ablation claims in §5.4 are based on this benchmark, the reported margins (e.g., Warp Error 0.683→0.553 and Temporal Consistency 0.665→0.710 in Table 2) may overfit the evaluation set. The authors should either fix hyperparameters on a validation split or report variance over multiple runs or splits.","section":"§5.4, Tables 3-6, Appendix B"},{"comment":"For the object-modification setting, Appendix A.1 states that the depth map is zeroed, the reference image is semantically unrelated to the source object, and the mask is a coarse rectangle. In this setting the source optical-flow prior is the only motion signal available to both MotRefNet and the evaluation metrics. The paper does not analyze the failure mode where the new object's motion differs from the source object, even though MotRefNet's warp guidance is then systematically wrong. A concrete test would be to edit objects with known different trajectories and measure flow consistency estimated on the edited video, or to compare with a variant that does not use source flow. Without such a test, the claim of 'robustly dealing with various editing objects' in §6 is not fully supported.","section":"Appendix A.1 and §4.2"}],"minor_comments":[{"comment":"The text refers to 'the forward diffusion schedule as in Equation (10)', but the forward diffusion formula is Equation (9); Equation (10) is the denoising loss. Please correct the cross-reference.","section":"§4.3, Eq. (10)"},{"comment":"There are frequent typographical inconsistencies: 'temoprally' in §1, 'M OTREFNET' and 'IVED IFF' vs 'IVEDiff' throughout, and 'Davis90' in §5.1 should be 'DAVIS' with the correct dataset version.","section":"Throughout"},{"comment":"Tables 3 and 6 are redundant; Table 6 should be the main table or Table 3 should be removed.","section":"Tables 3 and 6"},{"comment":"The qualitative comparisons are shown at low resolution without zoomed insets; at the printed size it is difficult for a reader to verify the claimed differences in temporal flicker and texture fidelity.","section":"Figures 4-9"},{"comment":"The paper does not report confidence intervals, standard deviations, or significance tests for any metric; with a benchmark of 236 triplets, several metrics (especially FID) are likely to be noisy.","section":"Tables 1-6"},{"comment":"Appendix A.2 reports that the model is trained on the Pexels dataset, but reference [23] points to Open-Sora-Plan rather than a Pexels data source; please clarify the data source and its license.","section":"Appendix A.2, reference [23]"}],"recommendation":"major_revision","confidential_remarks":"The paper does not mention code or model release, and the IVE-Benchmark is self-constructed; for a systems paper in this area, I would make public release of the benchmark annotations, trained checkpoints, and inference code a condition of acceptance. The claim of being 'the first' image-guided video editing diffusion model would benefit from a more systematic novelty check against concurrent work. The main technical concern is the shared source-flow dependence between the method and the evaluation metrics; this is addressable and not a fundamental flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nIVEDiff is a sensible first stab at image-guided video editing, assembled from known parts (MimicBrush + AnimateDiff-style inflation) with two genuinely new pieces: the optical-flow-guided MotRefNet and the masked-motion-modeling fine-tuning strategy. The writing is clear, the ablations are structured, and the gains over the three baselines on their own benchmark are consistent across Tables 1-6. That is real work and worth engaging with.\n\nThe soft spots are mostly about evaluation evidence, not about internal logic. The biggest is the temporal-consistency metrics. Warp Error and Temporal Consistency are computed by warping edited frames with optical flow extracted from the source video (Section 5.1), and MotRefNet is explicitly designed to follow that same source flow (Eqs. 3-6). So a model that reproduces source motion is rewarded even if the edited object's appearance is not faithfully propagated. The paper has no independent measure, e.g., flow estimated on the edited video, and no experiment where the edited object is expected to move differently from the source. That is a genuine evidence gap in the 'temporally smooth' claim; it is addressable, not fatal.\n\nSecond, the IVE-Benchmark is self-constructed, no error bars or significance tests are reported, and key MMM hyperparameters (mask ratio, stride, mask strategy) are selected on the same benchmark used for final evaluation. The differences between, say, 50% and 75% mask ratio in Table 6 are tiny (0.462 vs 0.464 Warp Error; 0.560 vs 0.560 Temporal Consistency), so without variance estimates these numbers are hard to trust as real differences.\n\nThird, the 'robustly dealing with various editing objects' claim is broader than the evidence. In object modification the depth is zeroed, the mask is a coarse rectangle, and the reference is semantically unrelated; source flow is the only motion prior. The failure mode where the new object moves differently from the original is not analyzed.\n\nNone of this undermines the core idea. The two proposed components are sensible, and the qualitative figures do show better temporal coherence than the baselines. For researchers working on video editing or temporal consistency in diffusion models, this is a useful task definition and baseline. The paper deserves a serious referee, but the evaluation needs an independent consistency metric, error bars, and ideally released code/benchmark before the strong claims are accepted.","headline":"A sensible first image-guided video editing system with two novel components, but the temporal-smoothness claim rests on a partly circular metric; still worth refereeing.","tokens_in":15938,"tokens_out":3478,"would_cite":true,"duration_ms":30598,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IVEDiff, the first image-guided video editing diffusion model, replaces masked regions with reference-image content while keeping the video temporally smooth.","keywords":["image-guided video editing","diffusion model","masked motion modeling","optical flow","temporal consistency","reference-based editing","motion module","video editing benchmark"],"falsifier":"Take a source video of an object moving left-to-right, mask it, and edit it with a reference image of a similar object that should move right-to-left. If the model outputs a video where the new object still moves left-to-right following the source flow, rather than adopting the reference's motion, then the warp prior is overriding the reference information and the claim of robust editing fails. Quantitatively, one could compare the edited video's optical flow inside the mask against both the source flow and the reference's implied motion.","tokens_in":14910,"feed_emoji":"🎬","tokens_out":7333,"duration_ms":63316,"temperature":0.7,"pith_summary":"This paper claims that image-guided video editing can be done with a diffusion model built on top of an existing image editor, without retraining the whole model from scratch. The authors introduce IVEDiff, which takes a source video, a reference image, and per-frame masks, and replaces the masked regions with semantically matching content from the reference while preserving temporal smoothness. Its two key ingredients are a motion reference network that uses optical flow from the source video to propagate edited frames, and a masked motion modeling fine-tuning strategy that trains only the newly added motion modules by partially occluding video frames. The paper also builds a benchmark of 236 video-mask-reference triplets and reports that IVEDiff lowers warp error and raises CLIP score relative to frame-wise editing and AnimateDiff-based baselines.","feed_headline":"One reference image edits an entire video","feed_subtitle":"IVEDiff replaces masked objects with reference content in every frame, keeping motion smooth and needing no text prompt.","key_machinery":"The two central objects are the optical-flow-guided motion reference network (MotRefNet) and the masked motion modeling (MMM) fine-tuning strategy. MotRefNet is inserted at each layer of the denoising UNet; it takes the latent features of two neighboring edited frames plus the optical flow computed by GMFlow from the corresponding source frames, predicts a two-channel per-pixel offset map, warps the previous edited frame's latents to align with the next frame, and then adds a learnable scaled residual to handle occlusion. MMM is the training scheme: from a video clip, the first frame is used as the reference image and the remaining frames are partially occluded by random grid masks, then the model is trained with the standard denoising objective while only MotRefNet and the motion module weights are updated and the base image editing model stays frozen. Together they let the model learn inter-frame motion dynamics without forgetting the intra-frame semantic editing capability of the base model.","core_discovery":"The central claim is that IVEDiff is the first image-guided video editing diffusion model and that it can generate temporally smooth edited videos while robustly handling a variety of editing objects at high quality. Given a reference image and per-frame masks, the model replaces the masked regions with content semantically related to the reference, and it does so without requiring text prompts. The paper shows that inflating the MimicBrush image editing model with a motion module and fine-tuning with masked motion modeling yields a better trade-off between inter-frame consistency and per-frame visual quality than frame-wise editing or naive AnimateDiff inflation. Quantitatively, on the object-modification task the full model reduces warp error from 0.683 to 0.553 and raises CLIP score from 23.43 to 25.84 relative to the strongest baseline.","pith_inferences":["The same warp-and-fine-tune recipe could plausibly extend to other guided video editing setups, such as text-guided or mask-guided inpainting, since the flow prior comes from the unedited source and does not depend on the modality of the reference.","If the source flow assumption is violated enough, the learned two-channel offset in MotRefNet would need to model the full motion difference between old and new content; the paper does not quantify when this correction saturates, so a stress test with motion-opposite references would be a natural next experiment.","The benchmark's metrics measure temporal and appearance statistics but not whether the edited object's movement is semantically consistent with the reference; a perceptual or motion-semantics metric would complement the reported numbers."],"forward_implications":["Image-guided video editing becomes a lightweight extension of image editing: inflating an existing image editor with a motion module and fine-tuning with MMM is enough to edit videos, avoiding full video-diffusion training.","Users can edit a video by pointing to an object and supplying a reference image, with no text prompt, lowering the interface barrier for localized video editing.","The IVE-Benchmark gives the field a shared set of 236 video-mask-reference triplets and five metrics (warp error, temporal consistency, FID, CLIP score) for comparing future methods.","Because unedited regions are preserved exactly, the same model can be applied iteratively to edit multiple objects in one video."],"supporting_citations":[{"why":"The base image-guided image editing model (MimicBrush) that IVEDiff inflates into a video editor; supplies the spatial editing and reference-UNet capability the paper preserves.","marker":"[6]"},{"why":"Supplies the motion-module design and inflation procedure that IVEDiff adapts, and provides the weights used to initialize the motion module.","marker":"[14]"},{"why":"The pre-trained optical flow predictor (GMFlow) that extracts the source-video flow priors fed into MotRefNet.","marker":"[43]"},{"why":"The denoising diffusion training objective used in the masked motion modeling fine-tuning.","marker":"[16]"},{"why":"The pre-trained depth estimator (Depth-Anything) that provides depth maps stacked into the editing UNet input.","marker":"[47]"}],"fun_headline_variants":["One image edits an entire video, no text needed","Text-free video editing with a single reference image","IVEDiff: image-guided video editing with smooth motion","Edit videos by showing a reference image"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that optical flow computed from the source video remains a valid propagation prior for the edited video, even when the masked region is replaced by semantically unrelated content.","fun_headline_variants_meta":{"raw":{"variants":["One image edits an entire video, no text needed","Text-free video editing with a single reference image","IVEDiff: image-guided video editing with smooth motion","Edit videos by showing a reference image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1599,"prompt_tokens":908,"completion_tokens":691,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":631}},"tokens_in":524,"tokens_out":691,"duration_ms":6455,"temperature":1.0,"reasoning_tokens":631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:35:48.416250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a source video of an object moving left-to-right, mask it, and edit it with a reference image of a similar object that should move right-to-left. If the model outputs a video where the new object still moves left-to-right following the source flow, rather than adopting the reference's motion, then the warp prior is overriding the reference information and the claim of robust editing fails. Quantitatively, one could compare the edited video's optical flow inside the mask against both the source flow and the reference's implied motion.","supporting_citations":[{"cited_title":"Gmflow: Learning optical flow via global matching","cited_arxiv_id":null,"evidence_quote":"The pre-trained optical flow predictor (GMFlow) that extracts the source-video flow priors fed into MotRefNet."},{"cited_title":"Denoising dif- fusion probabilistic models","cited_arxiv_id":null,"evidence_quote":"The denoising diffusion training objective used in the masked motion modeling fine-tuning."},{"cited_title":"Depth anything: Unleashing the power of large-scale unlabeled data","cited_arxiv_id":null,"evidence_quote":"The pre-trained depth estimator (Depth-Anything) that provides depth maps stacked into the editing UNet input."}],"review_version":1}