{"id":"f0a9f621-f6c3-494e-ac53-00bf848cb58e","arxiv_id":"2412.16153","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A motion-weighted training loss (MotiF) improves text alignment and object motion in text-image-to-video generation, winning 72% of human-preference comparisons against nine baselines on a new benchmark.","lead":"This paper proposes MotiF, a training tweak that reweights the video-generation loss toward pixels with strong motion, computed via optical flow, to make image-animation models follow text prompts better. The authors also release TI2V-Bench, a 320-pair benchmark with human-preference evaluation, on which MotiF beats nine open-sourced models with a 72% average win rate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Optical-flow magnitude conflates motion with text-relevant change; MotiF's headline improvement may reflect a flow-friendly benchmark and should be tested on appearance/new-object prompts.","rationale":"The paper is a clean, well-controlled method study: the ablation against w/o-MotiF with identical training is exactly the right experiment, and the human-evaluation protocol with justifications is a real asset. My concern is not that the loss does nothing; Figure 6 and Table 2 show it shifts learning to high-flow regions. The issue is scope of the central claim. MotiF is a motion-amplitude reweighting scheme, not a text-conditioned reweighting scheme; the text enters only through the underlying diffusion model. All the headline evidence is on TI2V-Bench, whose prompts were authored by the same team and predominantly ask for physical motions that produce optical flow. The paper itself acknowledges that new-object and multi-object prompts remain hard (Appendix C.3), and those are exactly the prompts where optical-flow magnitude is uncorrelated with text-relevant change. This does not falsify the method, but it means the headline 'improves text alignment' is only established for flow-dense motion instructions, not for TI2V generally. That warrants the reader's conditional verdict; a stratified evaluation would settle the question.","tokens_in":16304,"tokens_out":8817,"duration_ms":83859,"concrete_test":"Stratify TI2V-Bench (or an independent set) by whether the prompt's required change is flow-dense (e.g., 'car changing lane', 'horse jumping') vs flow-sparse (e.g., 'dolphin blowing bubbles', 'red light bulb flashing', 'frisbee enters scene'; also camera-only prompts). Re-run the controlled MotiF vs w/o-MotiF human A/B evaluation separately on each stratum, reporting text-alignment preference with confidence intervals. If the MotiF advantage is confined to flow-dense prompts and absent or reversed on flow-sparse prompts, the optical-flow proxy is not a general text-alignment mechanism and the central claim must be scoped; if the advantage persists on flow-sparse prompts, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is that L_motif = E||m'·(epsilon - epsilon_theta)||^2 reweights learning toward pixels with large optical flow, and the human-preference gains are attributed to this focus on text-driven motion. The load-bearing premise is that optical-flow magnitude identifies the pixels whose content the text specifies. That premise fails in three concrete regimes: (1) new objects entering the scene, where no correspondence exists and flow is near zero (the authors list this as a failure mode in Appendix C.3, Fig. A3c); (2) appearance/state changes such as 'red light bulb flashing' (Fig. A3d), where text-relevant change is photometric rather than translational; and (3) camera-driven videos, where global flow is high everywhere and the heatmap degenerates to a near-uniform weight, so it cannot isolate the object motion the text controls. The paper's own Appendix B.1 notes that flow-based dynamics metrics over-weight camera/background motion, which is exactly the confusion. Since TI2V-Bench was constructed by the authors with prompts that predominantly specify object translations, the 72% preference and the text-alignment justification may reflect better generation of flow-dense motions without establishing that MotiF improves text adherence for appearance/new-object instructions. The w/o-MotiF ablation (Table 2) controls for training, but it is evaluated on the same flow-friendly benchmark, so it does not test the proxy assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Motion Focal Loss (MotiF) for text-image-to-video (TI2V) generation. The method computes an optical-flow-derived motion heatmap from ground-truth training videos and reweights the standard diffusion denoising loss by this heatmap, so that the model concentrates on high-motion regions during training. The authors also introduce TI2V-Bench, a new benchmark of 320 image-text pairs across 22 scenarios, and a human evaluation protocol based on forced A-B preference with justification categories. They report that MotiF outperforms nine open-sourced TI2V methods with an average 72% human preference on TI2V-Bench, with the largest gains in text alignment and object motion.","tokens_in":16615,"tokens_out":4393,"duration_ms":39497,"significance":"If the claims hold, MotiF is a simple, training-time-only modification to the diffusion objective that is orthogonal to input-based motion priors and requires no extra conditioning at inference. The proposed benchmark and the justification-based human evaluation protocol are potentially useful resources for the community. The paper also provides a controlled ablation isolating the motion focal loss (Table 2) and discloses hyperparameter choices in the appendix. However, the significance depends on whether the optical-flow proxy for text-relevant motion is valid beyond flow-friendly scenarios, and on the statistical reliability of the reported human preferences.","major_comments":[{"comment":"The load-bearing premise of the method is that pixels with large optical flow are the pixels whose content is specified by the text prompt. This premise fails for appearance changes (e.g., 'the red light bulb flashing', Fig. A3d) and for new objects entering the scene (e.g., 'a dolphin blowing bubbles', Fig. A3c), where text-relevant change is photometric or involves no correspondence, and the flow-based heatmap assigns low weight to exactly those pixels. The paper itself lists these as failure cases in Appendix C.3, yet the headline 72% preference is aggregated over all prompts, including these difficult cases. To support the claim that MotiF improves text alignment, please report per-category or per-scenario preference results on TI2V-Bench (e.g., split by translation-type motion, appearance change, and new-object introduction), or evaluate on an additional benchmark enriched with such prompts, or extend the heatmap to incorporate photometric change (e.g., frame difference or segmentation masks). Without this, the central claim that the loss improves text-guided motion generation is not established for a substantial class of text-relevant changes.","section":"§3.2, Eq. (2), Appendix C.3, Fig. A3"},{"comment":"The human evaluation results are reported as preference percentages without confidence intervals, significance tests, or inter-annotator agreement statistics. With only 5 annotators per comparison and majority voting, the difference between MotiF and several baselines (e.g., 72% vs. 68.8% for VideoCrafter, and 72% vs. 72.5% for AnimateAnything) could be within noise. Please provide bootstrap confidence intervals, a paired significance test (e.g., Wilcoxon signed-rank on the per-item majority votes), or at least raw vote counts and agreement rates. This is essential because the central evidence for the method's effectiveness rests entirely on these human preference numbers.","section":"§5.2, Figure 4"},{"comment":"The motion focal loss weight λ was selected by evaluating on TI2V-Bench (Table A2), which is the same benchmark used for the headline comparisons in Figure 4. This creates a selection-bias risk: the reported 72% may be inflated by tuning λ on the test set. Please either fix λ a priori, or select it on a separate validation split and report the test results for that selection. At minimum, state explicitly how many hyperparameter configurations were tried and how the final λ=1 was chosen.","section":"Appendix A.2, Table A2"},{"comment":"The paper claims that MotiF 'complements existing techniques that utilize motion priors as model inputs.' However, the experiments only compare MotiF against released checkpoints of prior methods used out-of-the-box, without fine-tuning any baseline on the same training data or with the same image-conditioning setup. The claim of complementarity would require a direct experiment, e.g., adding MotiF to a motion-prior-based method or fine-tuning a strong baseline under identical conditions (same data, same x-cat injection). Without such an experiment, the complementarity claim is not supported by the presented evidence.","section":"§5.2, Table 4, and Section 6"}],"minor_comments":[{"comment":"The phrase 'we hypothesis that' should be 'we hypothesize that'.","section":"Section 6"},{"comment":"The phrase 'a dataset consists of 320 image-text pairs' is better as 'a dataset consisting of 320 image-text pairs' or 'a dataset that consists of 320 image-text pairs'.","section":"Abstract and Section 4.1"},{"comment":"The sentence 'We then apply a sigmoid-like function fl to normalize the intensity map' is awkward; 'fl' is the intensity map, so please rephrase to 'apply a sigmoid-like function to the intensity map fl'.","section":"Section 3.2, Motion Heatmaps"},{"comment":"The stacked bar chart is visually dense and difficult to compare across methods; consider a grouped bar chart or a table that separates the overall TI2V score from the justification percentages.","section":"Figure 4"},{"comment":"The training data is described only as an 'internal licensed dataset of 1M video-text pairs that is similar to [46]'. Since the dataset is not public, improving reproducibility would benefit from a more detailed description of the data composition, filtering, or licensing constraints.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The central concern is whether the optical-flow proxy is valid for the broad claim of improved text alignment; the paper's own failure cases (Appendix C.3) suggest this proxy misses important text-relevant changes. The self-built benchmark and the tuning of λ on that same benchmark (Table A2) add to the risk that the 72% result is partially an artifact of benchmark design. These issues are addressable within the scope of a major revision, so I recommend major revision rather than rejection. I would also encourage the editor to consider whether the evaluation should include a more diverse set of prompts beyond object translations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Couple of things to know. The core idea is genuinely simple: compute an optical-flow motion heatmap from ground-truth videos and use it to weight the diffusion loss, L_motif = E||m'·(eps - eps_theta)||^2, added to the standard L_diffusion. It works. The ablation in Table 2, w/o MotiF vs with, is properly controlled, and the inv-MotiF control strengthens the story. The paper is honest about failure modes: Appendix C.3 lists new-object entries, multiple objects, and appearance changes like a flashing bulb, and shows examples.\n\nWhat's new: reweighting the training objective rather than feeding motion into the model, plus TI2V-Bench, 320 image-text pairs across 22 scenarios. That benchmark may be useful to the community even if imperfect. The human evaluation is large: nine open-sourced models, majority vote of 5 annotators per comparison, justifications across four axes. The 72% average preference is a real result, not a fitted one — the loss derivation has no free parameters tuned to the benchmark, only lambda, which is set to 1.\n\nSoft spots. The headline evidence is human preference on a self-built benchmark without confidence intervals or significance tests. That matters because the benchmark is flow-friendly: many prompts describe translational motion (moving, jumping, rolling, turning), where optical flow is a good proxy for text-relevant change. The stress-test concern is real: flow magnitude misses new-object introductions and photometric changes (red bulb flashing), and the paper admits those are failure modes. So the 72% likely overstates the method's benefit for appearance-driven prompts. Ideally they'd add a separate human eval on prompts that are not flow-dense. Also, baselines are used out-of-the-box; a fairer comparison would give them the same training conditions or at least discuss that.\n\nThe math and citation pattern look fine. The loss is simple; the optical-flow heatmap comes from RAFT, a standard tool. Self-citation is not an issue here. Benchmark construction is described with enough detail to be replicable.\n\nWho this is for: researchers working on image-to-video or text-driven animation, and anyone building video evaluation benchmarks. It deserves a serious referee; the concerns are about evidence strength, not a load-bearing flaw. I'd recommend conditional acceptance, with the main revision being confidence intervals on the human eval and a secondary evaluation on appearance/new-object prompts, or at least a discussion of the proxy's limits.","headline":"A simple, well-tested loss reweighting for text-image-to-video that genuinely improves motion generation; the new benchmark is useful, but the headline human-preference evidence needs confidence intervals and a flow-blind follow-up.","tokens_in":17115,"tokens_out":2103,"would_cite":true,"duration_ms":18984,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MotiF reweights the diffusion loss by an optical-flow motion heatmap so the model concentrates on moving regions, and on a new 320-pair benchmark it beats nine open-sourced TI2V models with a 72% average human preference.","keywords":["text-image-to-video generation","image animation","motion focal loss","optical flow heatmap","diffusion model training","text alignment","human evaluation benchmark","TI2V-Bench"],"falsifier":"Curate a held-out set of prompts whose intended change is appearance-based rather than displacement-based (a light flashing, a balloon changing color, an object appearing), where ground-truth optical flow is near zero; if MotiF fails to improve text alignment over its baseline on that set, the flow-magnitude proxy is the failing link.","tokens_in":16142,"feed_emoji":"🎬","tokens_out":6249,"duration_ms":50389,"temperature":0.7,"pith_summary":"Text-guided image animation struggles because most of a video is static: in a typical training clip 97% of pixels do not move, so an equal-weight loss lets the model win by barely moving anything and ignoring the prompt. MotiF attacks this at the training objective: it computes an optical-flow heatmap of each video and multiplies the diffusion denoising error by that heatmap, forcing the model to spend its capacity on regions the text should be animating. On a new benchmark of 320 image-text pairs spanning 22 scenarios, with human A/B preference as the score, the approach beats nine open-sourced TI2V models with a 72% average preference, and the justifications show the gains are concentrated in exactly the target dimensions: text alignment and object motion. The paper also contributes TI2V-Bench and a justification-based human evaluation protocol for future comparisons.","feed_headline":"Motion-weighted loss wins 72% of human picks in image animation","feed_subtitle":"Weighting the training loss by optical flow makes text prompts drive motion, beating nine open models.","key_machinery":"The motion focal loss: a motion heatmap $m\\in[0,1]^{L\\times H\\times W}$ is built from RAFT optical flow between consecutive frames, normalized by a sigmoid $\\sigma(x)=1/(1+e^{100(0.05-x)})$ to polarize high- and low-motion pixels, downsampled to $m'$ to match the video latents, and multiplied elementwise into the standard noise-prediction MSE of Eq. (1) to give Eq. (2). This reweighting is what carries the argument: it converts the imbalance between static background and moving foreground into a learning signal, without requiring any extra input at inference time. The other load-bearing design choice is image conditioning by pure concatenation (x-cat) rather than cross-attention, which the paper argues avoids competition between image and text features.","core_discovery":"The central claim is that a simple per-pixel reweighting of the diffusion loss, $L_{\\text{motif}} = \\mathbb{E}_{t,x,\\epsilon}\\|m' \\cdot (\\epsilon - \\epsilon_\\theta)\\|_2^2$, where $m'$ is the optical-flow-derived motion heatmap downsampled to the latent grid, makes a TI2V model follow text-driven motion instead of collapsing toward the static first frame. Trained on top of a pretrained text-to-video model with the heatmap loss added at unit weight, the model wins 72% of pairwise human comparisons against nine open-sourced baselines on TI2V-Bench, and ablation against an inverse-motion reweighting shows that it is specifically the high-motion emphasis that helps. The paper further claims that concatenating the conditioning image with the noisy latent (x-cat) is the best image-injection choice, outperforming cross-attention alone and the combination, because cross-attention competes with text features.","pith_inferences":["Editorial inference: replacing the optical-flow heatmap with a text-conditioned saliency map (for example, from attention or segmentation) could extend MotiF to appearance-only changes such as flashes or new objects, which are the failure modes the paper reports.","Editorial inference: because the loss operates on the objective rather than the inputs, it should stack with conditional-image weakening and motion-prior injection; the gains could compound beyond what the 72% single-model result shows.","Editorial inference: the benchmark's finding that a static video tops image-alignment and text-alignment automatic scores suggests a motion-aware automatic metric is needed; one could score text-relevant motion regions rather than whole-frame similarity."],"forward_implications":["If the claim holds, any TI2V model trained with the motion-weighted loss should show improved text alignment and object motion with no extra inference-time inputs.","The loss is complementary to methods that feed motion scores or masks into the model, so combining both is a direct next step the paper leaves open.","The ablation with the inverse-motion weight implies that weighting the loss toward static regions actively hurts text-driven animation, so the direction of the reweighting is essential, not just the reweighting itself.","The TI2V-Bench protocol, with multiple prompts per image and forced-choice human justification, gives the community a single TI2V score plus per-axis diagnostics for future model comparisons.","On automatic metrics, MotiF is comparable to prior models, and the paper's static-video baseline result shows those metrics reward stillness; human preference is therefore the metric that reveals the improvement."],"supporting_citations":[{"why":"Supplies RAFT optical flow used to build the motion heatmap.","marker":"[42]"},{"why":"Provides the pretrained text-to-video diffusion model that MotiF adapts for TI2V.","marker":"[8]"},{"why":"Defines the dual-stream image-conditioning baseline and the dynamic frame-stride training recipe MotiF compares against.","marker":"[46]"},{"why":"Names the conditional image leakage problem and serves as a baseline that MotiF is designed to address.","marker":"[53]"},{"why":"Supplies AnimateBench, the existing benchmark on which the paper reports automatic metrics.","marker":"[51]"},{"why":"Guides the human A-B evaluation design with majority-vote preference.","marker":"[30]"},{"why":"Supplies the justification-based human evaluation protocol adapted for TI2V scoring.","marker":"[15]"}],"fun_headline_variants":["Motion-weighted loss makes text drive image animation","MotiF: motion focal loss improves text-motion alignment","72% human picks: MotiF wins with motion-weighted loss","Reweight loss by motion to animate images from text","MotiF beats nine open models by focusing on motion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the pixels with the largest optical flow in a training video are precisely the pixels the text prompt wants animated, which fails for flashes, color changes, and newly appearing objects.","fun_headline_variants_meta":{"raw":{"variants":["Motion-weighted loss makes text drive image animation","MotiF: motion focal loss improves text-motion alignment","72% human picks: MotiF wins with motion-weighted loss","Reweight loss by motion to animate images from text","MotiF beats nine open models by focusing on motion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000774,"raw_usage":{"total_tokens":3439,"prompt_tokens":974,"completion_tokens":2465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":2385}},"tokens_in":590,"tokens_out":2465,"duration_ms":16063,"temperature":1.0,"reasoning_tokens":2385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:44:50.157049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Curate a held-out set of prompts whose intended change is appearance-based rather than displacement-based (a light flashing, a balloon changing color, an object appearing), where ground-truth optical flow is near zero; if MotiF fails to improve text alignment over its baseline on that set, the flow-magnitude proxy is the failing link.","supporting_citations":[{"cited_title":"Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained text-to-video diffusion model that MotiF adapts for TI2V."},{"cited_title":"Dynamicrafter: Animating open-domain images with video diffusion priors","cited_arxiv_id":null,"evidence_quote":"Defines the dual-stream image-conditioning baseline and the dynamic frame-stride training recipe MotiF compares against."},{"cited_title":"Pia: Your personalized image animator via plug-and-play modules in text-to-image models","cited_arxiv_id":null,"evidence_quote":"Supplies AnimateBench, the existing benchmark on which the paper reports automatic metrics."},{"cited_title":"Emu video: Factoriz- ing text-to-video generation by explicit image conditioning","cited_arxiv_id":null,"evidence_quote":"Supplies the justification-based human evaluation protocol adapted for TI2V scoring."}],"review_version":1}