{"id":"515a3909-deba-47b8-b8ce-037d11fe7a55","arxiv_id":"2501.03059","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A mask-based motion trajectory representation plus masked attention improves multi-object image-to-video generation.","lead":"This paper presents a two-stage system for turning a single image into a video: it first predicts a per-object mask trajectory that captures motion, then uses that trajectory to guide a video diffusion model. The authors claim better multi-object motion consistency than existing open image-to-video models, and introduce a new benchmark for such tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage 2 is trained on ground-truth masks but evaluated on generated masks, and the paper never quantifies this gap; Stage-1 error propagation is an unmeasured load-bearing risk.","rationale":"I read the paper in good faith and find the two-stage design plausible, the ablations informative, and the TI2V control important for isolating the contribution of the masked-attention mechanism. The reader's conditional verdict is appropriate: the method is well motivated but not fully verifiable without code or data. My stress-test pass identifies a more specific, technically load-bearing gap than the general reproducibility concern: the Stage-2 model is trained with ground-truth mask trajectories but deployed with Stage-1 predictions, and the paper never measures how much this mismatch hurts. Because the strongest claim is about full-pipeline empirical performance, this unmeasured distribution shift directly threatens the central claim. The proposed oracle-versus-predicted experiment would settle whether Stage-1 errors are the bottleneck. I find no reason to move the verdict; it should remain conditional, pending this evidence.","tokens_in":17335,"tokens_out":7218,"duration_ms":76062,"concrete_test":"On SA-V-128 and Image-Animation-Bench, freeze the trained Stage 2 and run inference twice: (a) with the predicted Stage-1 trajectory \\hat{s}, and (b) with the ground-truth SAM2 trajectory s used during Stage-2 training. Report FVD, CLIPFrame, ViCLIP-T, ViCLIP-V, AD, and human preference for both. If (b) beats (a) by more than ~10% relative FVD or a large human-preference gap, Stage-1 error propagation dominates and the central claim is only conditional on Stage-1 accuracy; if the gap is small, the concern is resolved. Optionally, also report per-object mask mIoU of Stage 1 to localize the failure mode.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that mask-based motion trajectories yield state-of-the-art I2V generation, but the full pipeline is only as good as the Stage-1 mask predictions. In Sec. 3.3, the Motion-to-Video model is trained on the preprocessed tuple (x, s, c, cmotion, clocal), i.e., with the SAM2 ground-truth mask trajectory s. The masked-attention masks in Sec. 3.3.1 are likewise constructed from ground-truth s during training and only from generated \\hat{s} at inference. The paper reports no Stage-1 mask accuracy (e.g., per-object mIoU) and no oracle-versus-predicted comparison, so the magnitude of this train/test distribution shift is unknown. If Stage 1 misses an object, swaps identities, or drifts over time, Stage 2 receives out-of-distribution conditioning; the reported FVD/ViCLIP gains may then reflect the oracle-like training condition rather than the actual two-stage system. This is precisely the error-propagation path identified as the weakest assumption, and it is currently unmeasured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Through-The-Mask (TTM), a two-stage image-to-video (I2V) generation framework. Stage 1 (Image-to-Motion) generates a mask-based motion trajectory, i.e., a time-consistent per-frame semantic segmentation of motion-capable objects, conditioned on the first frame, its initial segmentation, and a motion-specific text prompt. Stage 2 (Motion-to-Video) generates the output video conditioned on the input frame, the full text prompt, and the predicted mask trajectory, injecting object-level information via two proposed masked attention objectives: masked cross-attention that binds object-specific prompts to the corresponding latent regions, and masked spatio-temporal self-attention that constrains attention to same-object positions for temporal consistency. The method is evaluated on a new benchmark SA-V-128 (64 single-object and 64 multi-object videos) and on Image-Animation-Bench, with U-Net and DiT backbones, reporting improvements in FVD, CLIPFrame, ViCLIP-T/V, and human preference over several baselines, plus ablations showing the contribution of each masked attention component and a comparison against optical flow as the intermediate representation.","tokens_in":17542,"tokens_out":6205,"duration_ms":56527,"significance":"If the empirical results are reliable, this is a solid contribution: the paper demonstrates that a compact object-level intermediate representation (mask trajectories) can be more effective than per-pixel optical flow for controllable I2V generation, and the proposed masked attention objectives are architecture-agnostic, being validated on both U-Net and DiT. The ablations in Table 3 and Table 4 are well designed to isolate the contribution of each component, and the comparison against optical flow is a meaningful and non-trivial empirical finding. The main caveat is that the full two-stage system is evaluated without measuring the gap between training-time conditioning on ground-truth masks and inference-time conditioning on predicted masks; this is load-bearing for the central SOTA claim. The lack of error bars and the author-curated new benchmark further moderate the strength of the conclusions. The paper does not ship code or data, which limits reproducibility, but the method description and ablations are detailed enough to permit re-implementation.","major_comments":[{"comment":"Stage 2 is trained on the tuple (x, s, c, cmotion, clocal) with ground-truth mask trajectories s, and the masked attention masks M^(l) and M_self in Eqs. (1)-(3) are constructed from ground-truth segmentation during training. At inference, however, the model conditions on the predicted trajectory \\hat{s} generated by Stage 1, and the attention masks are derived from \\hat{s}. The paper reports no metric on Stage-1 mask quality (e.g., per-object mIoU against the ground-truth trajectory, temporal consistency) and no oracle-vs-predicted comparison for Stage 2. Consequently, the reported gains (e.g., Table 1 FVD 925 vs. 1089) could partly reflect the oracle-like training condition rather than the actual two-stage system. I ask the authors to (i) report Stage-1 mask accuracy on the evaluation benchmarks, and (ii) run Stage 2 with ground-truth masks and with predicted masks (and ideally a no-mask-conditioning baseline) to quantify the error-propagation gap. This is necessary to support the claim that mask trajectories are a robust intermediate representation.","section":"Sec. 3.3 and 3.3.1"},{"comment":"All reported automatic metrics are point estimates without confidence intervals or significance tests, and the number of generated videos per cell is not stated for the Image-Animation-Bench results. Several differences between the proposed method and baselines are small (e.g., Table 3 ViCLIP-T 0.220 vs. 0.220; Table 1 ViCLIP-T 0.220 vs. 0.220 vs. 0.218). Without bootstrapped confidence intervals or repeated-seed evaluations, the 'state-of-the-art' claim is not fully supported. Please provide confidence intervals for FVD, CLIPFrame, ViCLIP-T, and ViCLIP-V and at least an inter-rater agreement measure for the human evaluation.","section":"Tables 1-4"},{"comment":"The mask-vs-flow ablation compares two full two-stage systems, but the authors do not report the accuracy of the Stage-1 predictors for each representation. It is therefore unclear whether the large FVD difference (648 vs. 1014) is caused by the representation itself or by the relative difficulty of generating accurate flow compared to accurate masks, or by the train/test shift described in the first comment. An oracle comparison (feeding ground-truth flow vs. ground-truth masks directly into Stage 2, avoiding Stage-1 error) would isolate the effect of the representation on the generator and would strengthen the paper's central claim about the choice of intermediate representation.","section":"Sec. 4.3 and Table 4"},{"comment":"The new benchmark SA-V-128 is created by the authors via an automatic selection process: videos are scored by aesthetic quality and optical-flow-based motion, the top 500 are kept, and then 64 single-object and 64 multi-object videos are randomly drawn. Since this benchmark is a primary testbed for the SOTA claim, the authors should release the exact list of videos, the selection script, and ideally the scores, and report the stability of the metric rankings across multiple random draws of the final 128. Without such details, the possibility that the benchmark is accidentally biased toward the method's strengths cannot be ruled out.","section":"Sec. 4.1 (SA-V-128)"}],"minor_comments":[{"comment":"Cell values in Table 1 and Table 2 suffer from malformed spacing, e.g., '8.9475.0' and '4.8778.9'; these should be fixed to clearly separate the two numbers.","section":"Tables 1-2"},{"comment":"The Image-Animation-Bench is used as an evaluation benchmark, but no original citation or description of its curation is given in the main text; please cite the source or provide a reference to the supplementary material that defines it.","section":"Sec. 4.1"},{"comment":"The Average Displacement (AD) metric is reported without the corresponding AD of the ground-truth reference videos; without this reference value, it is difficult to interpret whether the reported AD indicates realistic motion or merely a specific motion magnitude range.","section":"Sec. 4.1 (metrics)"},{"comment":"In the ablation table, the 'no mask attn' configuration still concatenates the mask trajectory as an input to the denoising network, whereas the TI2V baseline does not use a mask trajectory at all; the text should state this difference explicitly to avoid confusion about what each row represents.","section":"Sec. 4.3 and Table 3"},{"comment":"The notation \\hat{s}_\\theta(s_t, t, E(x(0)), E(s(0)), c_motion) is ambiguous because s_t denotes the noised trajectory and the output is the predicted clean trajectory; please clarify that during training s_t is the noised ground-truth trajectory and during inference the model is conditioned on the noisy latent.","section":"Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from a strong industrial lab and the experiments are extensive, but the evaluation is entirely internal (Meta datasets, in-house baselines). The new benchmark is derived from SA-V, also a Meta dataset, and the selection criteria may inadvertently favor the method. Given the lack of released code, the unmeasured train/test gap in mask conditioning, and the absence of error bars, I would recommend insisting on the oracle-vs-predicted analysis and a stability analysis of the benchmark before acceptance. These are fixable but require additional experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. The core idea—using a mask-based motion trajectory as an explicit intermediate representation for image-to-video instead of optical flow—is genuinely useful, and the paper backs it with clean ablations. The two masked attention mechanisms (per-object cross-attention and object-restricted self-attention) are a sensible extension of masked conditioning from text-to-image to video, and the ablations show each contributes on both U-Net and DiT backbones. The mask-vs-flow comparison is the most convincing part: masks win by a large margin on FVD and frame consistency, and the qualitative examples show flow collapsing where masks don't. That is a real result. The paper is honest about what it builds on (SpaText for masked cross-attention, Motion-I2V for two-stage) and does not oversell novelty. Credit also for reporting ablations on two architectures and for a new benchmark that separates single- and multi-object cases.\n\nSoft spots, in order of real weight. First, the train/test mismatch the stress test flags is valid: Stage 2 trains on ground-truth SAM2 masks but infers on Stage-1 predicted masks, and the paper never measures Stage-1 mask accuracy or reports an oracle-vs-predicted comparison. If Stage 1 drops an object or drifts over time, Stage 2 receives conditioning it never saw in training. The full system does produce better videos than the baselines, so Stage 1 is probably doing something right, but that \"probably\" should be measured. This is fixable with a simple experiment. Second, there are no error bars or significance tests, and several gaps are small—ViCLIP-T differences of 0.001–0.004 could easily be noise. Third, SA-V-128 is author-curated from SA-V using a selection procedure that favors high motion and aesthetics; not disqualifying, but it is a curated set with no external validation. Fourth, the paper claims SOTA but compares against a limited set of baselines (VideoCrafter, DynamiCrafter, Motion-I2V, ConsistI2V, and their own TI2V); notable recent I2V models are absent, which is fine for a workshop-style claim but weaker for a SOTA claim.\n\nWho this is for: anyone working on I2V or controllable video generation. The mask-trajectory representation and the masked-attention recipe are worth serious attention, and the paper is well written enough to follow and build on. It deserves a serious referee. I would send it to review and ask for the Stage-1 oracle comparison, error bars, and code/data release before accepting.","headline":"A solid, well-ablated I2V paper whose mask-based motion trajectories beat optical flow, but the unmeasured Stage-1-to-Stage-2 gap and missing code/data keep it from being fully settled.","tokens_in":18059,"tokens_out":1809,"would_cite":true,"duration_ms":17856,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that image-to-video generation becomes more temporally coherent and text-faithful, especially with multiple objects, when motion is first represented as per-object mask trajectories and injected into a video diffusion…","keywords":["image-to-video generation","mask-based motion trajectory","two-stage video generation","masked cross-attention","masked self-attention","multi-object motion","diffusion models","video segmentation"],"falsifier":"Take a trained system and systematically delete one object's mask from the generated trajectory at inference, then measure the deleted object's motion in the output video. If the video still animates that object correctly, the mask trajectory is not carrying the motion and the two-stage claim is false. A complementary check is comparing mask-based and optical-flow-based stage-2 conditioning on a benchmark with many small, fast objects, where mask-level trajectories that lose such objects should show clear per-object motion failures.","tokens_in":17174,"feed_emoji":"🎬","tokens_out":7753,"duration_ms":59524,"temperature":0.7,"pith_summary":"Image-to-video models must animate a still photo according to a text prompt, and they often fail when several objects have to move and interact. This paper argues that the failure comes from forcing one network to infer object semantics, motion, and appearance all at once, and that a cleaner division of labour is possible. The proposed method first generates an explicit mask-based motion trajectory—a time-consistent per-object segmentation of the scene—and then lets a standard video diffusion model render the video while masked attention objectives tie each object to its own prompt and its own moving region. The authors claim this two-stage decomposition yields state-of-the-art temporal coherence, motion realism, and prompt faithfulness on multi-object benchmarks, and that masks beat optical flow as the intermediate motion representation.","feed_headline":"Segmentation masks steer video generation better than pixel flow","feed_subtitle":"Coarse per-object masks carry motion and semantics, fixing multi-object drift in generated video.","key_machinery":"The load-bearing object is the mask-based motion trajectory: a sequence of per-frame binary masks, color-coded by object, that tracks each motion-capable object through time and doubles as a semantic label. Stage 1 predicts this trajectory from the reference image, its initial SAM2 segmentation, and a motion-only text prompt. Stage 2 conditions video synthesis on the trajectory through masked cross-attention and masked spatio-temporal self-attention, both built by adding log-masks to the attention logits: $h_{\\mathrm{cross}} = \\sigma(qk^\\top/\\sqrt{d} + \\log M_{\\mathrm{cross}})v$ and $h_{\\mathrm{self}} = \\sigma(qk^\\top/\\sqrt{d} + \\log M_{\\mathrm{self}})v$. These masks are what force the generator to keep each object's appearance in its own region and to keep each object consistent across frames.","core_discovery":"The central claim is that a coarse, semantic, per-object mask trajectory is a better intermediate representation for image-to-video than either end-to-end latent mapping or pixel-level optical flow. The mask trajectory expresses both what each object is and where and how it moves, while deliberately discarding precise per-pixel motion; the second stage then fills in appearance and fine detail. To make the masks causally effective, the method injects them through two attention mechanisms: masked cross-attention, which lets each spatial latent position attend only to the text prompt of the object occupying that position, and masked self-attention, which lets each position attend only to positions of the same object across frames. In the paper's experiments, this configuration outperforms existing image-to-video methods and the authors' own matched baselines on FVD, CLIP-based faithfulness, and human preference.","pith_inferences":["This is an editorial extension: because the trajectory is an explicit mask sequence, a user could drag, delete, or re-color an object's mask and re-run stage 2, steering the video without retraining the diffusion model.","A natural next step is text-to-video and video editing, where an LLM or detector proposes object masks from text alone, removing the need for a reference image at stage 1.","The paper's logic predicts localized errors: if stage 1 misses an object, only that object's motion should degrade, so per-object motion metrics would be a sharper test than whole-video FVD."],"forward_implications":["If the central claim holds, multi-object image-to-video becomes inspectable: the intermediate mask trajectory can be examined and edited before video rendering, instead of trusting the model to infer motion from the prompt alone.","The two-stage recipe is architecture-agnostic, demonstrated on both U-Net and DiT backbones, so the same mask-conditioning objectives can be added to existing video diffusion models.","Replacing optical-flow conditioning with mask conditioning shifts the hard part of the task: the first stage predicts coarse object-level motion, while the second stage keeps pixel-level appearance.","Masked self-attention is the mechanism that ties each object's appearance to its own region across frames, which is where the reported FVD and ViCLIP-V improvements concentrate."],"supporting_citations":[{"why":"The closest two-stage baseline; it uses optical flow as the intermediate motion representation that this paper replaces with masks.","marker":"[43]"},{"why":"Supplies the masked cross-attention formulation for region-specific text conditioning, extended here from image to video generation.","marker":"[3]"},{"why":"SAM2 performs the video segmentation that produces ground-truth mask trajectories for training and the initial mask at inference.","marker":"[40]"},{"why":"Provides the object-level masked cross-attention mechanism adapted into the video setting for integrating object-specific prompts.","marker":"[33]"},{"why":"Gives the latent diffusion formulation used in both the image-to-motion and motion-to-video stages.","marker":"[42]"},{"why":"Movie Gen is the DiT backbone the paper adapts for the motion-to-video stage, demonstrating architecture-agnostic gains.","marker":"[38]"},{"why":"AnimateDiff is the U-Net backbone adapted for the image-to-motion and motion-to-video stages.","marker":"[17]"},{"why":"Grounding DINO supplies object bounding boxes in the first frame that seed the SAM2 video segmentation during data pre-processing.","marker":"[29]"}],"fun_headline_variants":["Mask trajectories beat optical flow for image-to-video","Per-object mask motion improves multi-object video generation","Two-stage I2V: mask-based motion then attention refinement","From still to video: mask motion trajectories as the key","Coarse mask trajectories capture semantics and motion for I2V"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline depends on the first stage producing a correct, complete set of per-object masks from a single image and a text prompt: if an object is missed or its mask drifts, the second stage cannot recover that object's motion, because the attention masks only tell it where to look.","fun_headline_variants_meta":{"raw":{"variants":["Mask trajectories beat optical flow for image-to-video","Per-object mask motion improves multi-object video generation","Two-stage I2V: mask-based motion then attention refinement","From still to video: mask motion trajectories as the key","Coarse mask trajectories capture semantics and motion for I2V"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000645,"raw_usage":{"total_tokens":2980,"prompt_tokens":977,"completion_tokens":2003,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":1923}},"tokens_in":593,"tokens_out":2003,"duration_ms":71233,"temperature":1.0,"reasoning_tokens":1923,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:56:35.008536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained system and systematically delete one object's mask from the generated trajectory at inference, then measure the deleted object's motion in the output video. If the video still animates that object correctly, the mask trajectory is not carrying the motion and the two-stage claim is false. A complementary check is comparing mask-based and optical-flow-based stage-2 conditioning on a benchmark with many small, fast objects, where mask-level trajectories that lose such objects should show clear per-object motion failures.","supporting_citations":[{"cited_title":"Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling, 2024","cited_arxiv_id":null,"evidence_quote":"The closest two-stage baseline; it uses optical flow as the intermediate motion representation that this paper replaces with masks."},{"cited_title":"Spatext: Spatio-textual representation for con- trollable image generation","cited_arxiv_id":null,"evidence_quote":"Supplies the masked cross-attention formulation for region-specific text conditioning, extended here from image to video generation."},{"cited_title":"Compositional text-to-image gen- eration with dense blob representations, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the object-level masked cross-attention mechanism adapted into the video setting for integrating object-specific prompts."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Gives the latent diffusion formulation used in both the image-to-motion and motion-to-video stages."},{"cited_title":"Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petro- vic, and Yuming Du","cited_arxiv_id":null,"evidence_quote":"Movie Gen is the DiT backbone the paper adapts for the motion-to-video stage, demonstrating architecture-agnostic gains."},{"cited_title":"Animatediff: Animate your personalized text-to- image diffusion models without specific tuning, 2024","cited_arxiv_id":null,"evidence_quote":"AnimateDiff is the U-Net backbone adapted for the image-to-motion and motion-to-video stages."}],"review_version":1}