{"id":"464267f2-c3bf-4911-a37d-a21d8776c097","arxiv_id":"2412.01343","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MoTrans transfers specific motions from reference videos to new subjects using a two-stage fine-tuning scheme with recaptioned prompts, appearance injection, and a motion-specific verb embedding.","lead":"This paper presents MoTrans, a method that teaches a text-to-video AI model a specific motion from one or a few reference videos, then generates new videos of different subjects performing that same motion in new scenes. It is a step toward practical, customizable video generation for animation and film, letting creators reuse a motion without retraining the whole model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MoFid is unvalidated and likely confounded by appearance; the quantitative support for the motion-transfer claim is therefore not yet established.","rationale":"The reader's conditional verdict is reasonable, and I would not downgrade it. However, the most load-bearing weakness is not the Eq. (2) notation ambiguity, which is a reproducibility concern rather than a direct attack on the empirical claim. The stronger issue is that MoFid, the only quantitative metric specifically intended to measure motion fidelity, is self-defined and unvalidated, and the paper's own discussion in Section B.2 acknowledges that MoFid is sensitive to appearance. This makes the quantitative comparison in Table 1 and the ablation study in Table 2 ambiguous when used as evidence for the motion-transfer claim. The user study asks the right question, motion similarity 'without resembling appearance', and MoTrans wins it, which is genuine independent support. But the user study is a relative preference and does not calibrate MoFid. The concrete control experiment would settle whether MoFid isolates motion. If it does, the conditional accept is clearly justified; if it does not, the central claim still has qualitative and user-study support, so the verdict should remain conditional rather than reject, pending a validated motion metric or release of code and data.","tokens_in":15561,"tokens_out":7630,"duration_ms":71240,"concrete_test":"Run a control experiment on MoFid using the same reference videos: (i) create an appearance-matched, motion-free video by repeating a random frame from each reference video; (ii) create a motion-matched, appearance-different video by driving a different subject (e.g., a synthetic skeleton or a different identity) through the same motion, or by using a model known to copy motion without appearance. Compute MoFid for both sets. A valid motion-fidelity metric should score (ii) as high as genuine transfer videos and (i) much lower. If (i) scores comparably to the reported MoTrans values, or (ii) scores low, MoFid is dominated by appearance and the quantitative motion comparisons in Tables 1 and 2 are inconclusive. As a secondary check, report MoFid and CLIP-E over at least three random seeds or reference selections with standard deviations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MoTrans learns the specific motion pattern from reference videos and transfers it to new subjects without appearance overfitting. The only quantitative metric aimed at motion is MoFid (Eq. 8), which averages VideoMAE cosine similarity between generated videos and one randomly selected training video per motion. This metric is confounded: VideoMAE embeddings encode appearance as well as motion, and the paper itself concedes in Section B.2 that a high MoFid with low CLIP-T/CLIP-E 'typically indicates' appearance overfitting. That concession means MoFid responds to appearance similarity, yet no calibration or control experiment shows it isolates motion. Consequently, Table 1's key motion comparison is ambiguous: MoTrans's one-shot MoFid (0.5679) is actually below ZeroScope-finetune (0.6011), and the paper's explanation, that the finetuned model overfits appearance, depends on reading MoFid as a motion signal while simultaneously treating its appearance sensitivity as a defect. The ablation study has the same problem: removing the appearance injector raises MoFid (0.6150 vs 0.5679), which is interpreted as overfitting, but could equally mean the injector reduces motion fidelity. Without a validated motion metric, the central claim rests on the user study and qualitative examples, which are encouraging but do not quantitatively establish motion transfer.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MoTrans, a two-stage method for customizing motion transfer with text-driven video diffusion models. In an appearance learning stage, an MLLM-based recaptioner expands the prompt and spatial LoRAs are trained; in a motion learning stage, an appearance injector pre-injects frame embeddings into the UNet and temporal LoRAs are trained together with a motion-specific residual embedding. The method supports one-shot and few-shot motion customization and is evaluated on a self-collected 12-motion benchmark against ZeroScope-finetune, Tune-a-Video, MotionDirector, LAMP, and DreamVideo using CLIP-T, CLIP-E, TempCons, and a newly proposed MoFid metric, plus a user study. The central claim is that MoTrans learns the specific motion from reference videos and transfers it to new subjects while avoiding appearance overfitting.","tokens_in":15840,"tokens_out":3499,"duration_ms":32339,"significance":"If the central claim holds, MoTrans is a practically useful contribution: it addresses appearance-motion decoupling in a minimally invasive way (LoRAs only, no per-frame pose control) and supports both single- and multi-video customization, including simultaneous subject and motion customization. The paper ships a relatively complete experimental package with qualitative comparisons, ablations, and a user study. However, the quantitative support for the motion-transfer claim depends on a newly introduced metric, MoFid, that is not validated against human judgment or shown to isolate motion from appearance. The absence of error bars or significance tests further weakens the numerical comparisons. The method itself is clearly described conceptually, but one core equation is inconsistent with its textual description, making the exact mechanism ambiguous. These issues are fixable but currently prevent the evidence from fully supporting the stated claims.","major_comments":[{"comment":"The text states that the linearly projected image embedding is 'summed' with the hidden states before the temporal transformer, but Eq. (2) uses the broadcast operator ⊙, which conventionally denotes element-wise product. This is a load-bearing ambiguity: an additive injection and a multiplicative injection are different mechanisms and would have different effects on how appearance information interacts with temporal layers. Please correct the equation or the text, specify the exact implementation, and, ideally, provide an ablation comparing additive vs. multiplicative injection to confirm that the reported behavior is due to the intended operation.","section":"Section 3.2, Eq. (2)"},{"comment":"The proposed MoFid metric is not validated and appears confounded by appearance. The paper's own analysis in B.2 says that a high MoFid with low CLIP-T/CLIP-E 'typically indicates an overfitting to the reference's appearance,' which means MoFid responds to appearance similarity as well as motion. No control experiment is provided to show that MoFid isolates motion (e.g., same subject with different motions, different subjects with the same motion, or correlation with human motion-similarity judgments). Consequently, the key quantitative comparison in Table 1 is ambiguous: for one-shot, ZeroScope-finetune achieves a higher MoFid (0.6011) than MoTrans (0.5679), and the paper attributes this to appearance overfitting, but without a validated metric this could also indicate higher motion fidelity. Please validate MoFid or supplement it with a human-validated motion fidelity evaluation before using it as the primary evidence for the motion-transfer claim.","section":"Section 4.1, Eq. (8), and Section B.2"},{"comment":"All quantitative results are reported as point estimates without error bars, multiple seeds, or significance tests. Given that several differences are small (e.g., one-shot MoFid 0.5679 for MoTrans vs. 0.5627 for Tune-a-Video; ablation MoFid 0.5679 vs. 0.5643 without motion enhancer), the reported improvements may be within run-to-run noise. Please report mean and variance over at least three seeds or reference-video subsets, and apply an appropriate significance test for the main comparisons against the strongest baselines.","section":"Tables 1, 2, and 3"},{"comment":"The appearance injector randomly selects one image embedding from the reference video(s) at each training step. When multiple reference videos are provided for few-shot learning, the text does not specify whether the random selection is uniform across frames within a single randomly chosen video, or across all videos. This lack of detail affects reproducibility and could also introduce unintended variance in the learned motion representation. Please specify the sampling procedure precisely.","section":"Section 3.2, Appearance injector"}],"minor_comments":[{"comment":"Equation (1) refers to a VQ-VAE for compressing frames, but ZeroScope, the stated base model, uses a latent diffusion VAE rather than a VQ-VAE. Please correct the terminology.","section":"Section 4.1, Implementation details"},{"comment":"The notation |¯v_m| is ambiguous; it is later used as the count of generated videos for motion m. Please define it explicitly, for example as N_m, to avoid confusion with set cardinality or absolute value.","section":"Section 4.1, Eq. (8)"},{"comment":"The user study reports 1536 answers from 32 participants, but no statistical significance testing is reported, and the recruitment process is not described. A paired comparison test (e.g., Wilcoxon signed-rank) over the per-participant preferences would strengthen the claim that MoTrans is preferred over each baseline.","section":"Section 4.2, User study"},{"comment":"The benchmark is self-constructed and the paper does not mention whether the dataset or the evaluation code will be released. For reproducibility of the quantitative comparisons, please state the release plan or provide full details of the collected videos and prompt templates.","section":"Appendix A"},{"comment":"Figure 9 is referenced in the supplementary analysis but appears to be a scatter plot without axis labels or a clear description of what each point represents. Please add axis labels and a caption explaining the plotted points (e.g., one point per method per motion or per generated video).","section":"Section B.2 and Figure 9"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be the camera-ready version of an ACM Multimedia 2024 paper. If it is being considered for journal publication, the authors should clarify what has been added beyond the conference version, as the present manuscript reads as a conference paper in scope and depth. The core idea is promising, but the quantitative validation currently depends on an unvalidated metric and lacks statistical rigor; these need to be addressed before the claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a real, workmanlike approach to motion customization, and the central claim is probably right. The two-stage design (spatial LoRAs for appearance, temporal LoRAs plus a verb-conditioned residual embedding for motion) is a sensible combination of existing ideas, and the ablations show each piece does something. The user study directly asks about motion similarity without appearance leakage, and MoTrans wins there. That is genuine evidence.\n\nWhat's genuinely new is the motion-specific residual embedding computed from the pooled video embedding and concatenated with the verb embedding, and the MoFid metric. The MLLM recaptioner and appearance injection are more incremental (DreamVideo does something similar), but the paper frames them clearly.\n\nNow the soft spots. MoFid is the load-bearing quantitative metric for motion fidelity, and it is not validated. The paper's own Section B.2 says high MoFid with low CLIP scores 'typically indicates an overfitting to the reference's appearance.' That concession means MoFid responds to appearance similarity, not just motion. In Table 1, the one-shot ZeroScope-finetune MoFid (0.6011) is actually higher than MoTrans (0.5679); the authors explain that as overfitting, but that interpretation relies on reading MoFid as a motion signal while simultaneously knowing it is appearance-sensitive. The ablation results have the same ambiguity: removing the appearance injector raises MoFid (0.6150 vs 0.5679), which could mean the injector hurts motion fidelity rather than prevents overfitting. Without a validated motion metric, the quantitative comparison is weak.\n\nMinor issues: Equation (2) says the appearance embedding is 'summed' with the hidden states in the text, but the equation uses a broadcast product (⊙). Fixable typo, but it matters. No error bars or significance tests anywhere, which for a user study with 32 participants is a real gap. No code or data released. The comparison omits two baselines cited in the related work (Customize-A-Video and MotionCrafter); the explanation for DreamVideo is reasonable (they had to use the original ModelScope T2V), but the omission is worth noting.\n\nOverall: the paper is worth engaging with. The method is plausible, the ablations are honest, and the qualitative results are decent. But the quantitative evidence for the core claim is not airtight because the proposed metric is confounded. This is exactly the kind of paper that should go to peer review — it will be improved by referees pushing on the metric and asking for a released benchmark. I'd accept it as a CONDITIONAL, and I'd want to see a calibration study for MoFid before trusting the tables.","headline":"Solid motion-customization recipe with a plausible central claim, but the proposed MoFid metric is confounded by appearance and the quantitative evidence is weaker than the user study.","tokens_in":16354,"tokens_out":3455,"would_cite":true,"duration_ms":27314,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoTrans claims that a text-to-video diffusion model can learn a specific human-centric motion from one or a few reference videos and transfer it to new subjects and scenes without copying the reference appearance.","keywords":["motion customization","text-to-video diffusion models","appearance-motion decoupling","low-rank adaptation","multimodal recaptioning","motion-specific embedding","video generation","customized video generation"],"falsifier":"Train MoTrans on two reference sets with the same motion but very different appearances (for example, a human lifting weights and a fluffy teddy bear lifting weights), generate videos of a new subject such as a tiger, and compare the outputs for appearance differences beyond sampling noise. If the tiger's texture or identity systematically changes with the reference set, the appearance injector is leaking appearance and the decoupling claim fails.","tokens_in":15379,"feed_emoji":"🎬","tokens_out":13576,"duration_ms":99721,"temperature":0.7,"pith_summary":"MoTrans claims that a text-to-video diffusion model can learn a specific, complex human motion from as little as one reference video and then perform that motion on new subjects in new scenes, without inheriting the reference video's appearance. The proposal is a two-stage fine-tuning scheme: a first stage trains spatial LoRA adapters to absorb appearance from an MLLM-expanded prompt, and a second stage trains temporal LoRA adapters to absorb motion while a projected image embedding is injected to keep appearance out of the motion path. A residual embedding computed from the video and the verb token sharpens the motion concept. The authors report that this beats existing motion-customization methods on text alignment, entity alignment, temporal consistency, and their proposed motion-fidelity metric, with user-study results in the same direction. If correct, it gives animators and filmmakers a lightweight way to reuse complex human-centric actions on arbitrary characters.","feed_headline":"One reference video is enough to transfer a motion to new subjects","feed_subtitle":"Separate training stages for look and movement let a panda skateboard or a tiger lift weights.","key_machinery":"The central mechanism is a two-stage LoRA-based decoupling pipeline inside a pretrained text-to-video UNet. In the appearance stage, the MLLM-based recaptioner expands the user prompt to describe the foreground subject and background, and only spatial self-attention and feed-forward LoRAs are updated. In the motion stage, the appearance injector takes a random frame from the reference video, encodes it with an OpenCLIP image encoder, passes it through a linear layer, and broadcasts the result into the spatial transformer's hidden states before the temporal transformers, so the temporal LoRAs are steered toward motion; concurrently, a motion enhancer MLP produces a residual embedding from the mean-pooled video embedding and the verb's text embedding, which is added to the verb token and regularized with an L2 term. These pieces together are what the paper claims separates what the video looks like from what the video does.","core_discovery":"The core claim is that appearance and motion in a small reference video set can be decoupled by giving the model two complementary appearance signals—an MLLM-expanded textual description and a visual embedding of a random frame—while reserving a separate residual embedding to represent the motion itself. Concretely, the method trains spatial LoRAs on the recaptioned prompt, freezes them, then trains temporal LoRAs while broadcasting a linearly projected image embedding into the UNet hidden states just before the temporal transformers, compelling those layers to model only dynamics. The motion enhancer locates the verb in the prompt, concatenates the mean-pooled video embedding with the verb's text embedding, and forms a residual embedding added to that token, so the specific motion is represented at the conditioning level. The paper argues this two-stage multimodal decoupling solves the overfitting failure of prior fine-tuning approaches and achieves superior performance in one-shot and few-shot motion transfer.","pith_inferences":["A consequence the paper does not state is that the appearance injector's broadcast weight acts as a dial between motion fidelity and subject alignment; a direct test would sweep this scale and measure the MoFid/CLIP-E trade-off.","Because the motion enhancer depends on locating the verb via part-of-speech tagging, the method is sensitive to prompt phrasing; an unstated extension is testing whether the residual embedding still helps when the prompt has no explicit verb or contains several motion verbs.","The frozen spatial LoRAs open a path toward composable customization: an independently trained motion LoRA from one reference set could be paired with a spatial LoRA from another subject, something the paper demonstrates with exemplar images but does not test for arbitrary independently trained adapters.","If the decoupling mechanism is robust, the specific image encoder used for the appearance injector may not be essential; swapping OpenCLIP for another frame encoder and re-measuring MoFid and CLIP-E would probe how much of the decoupling is due to the architecture versus the encoder's inductive bias."],"forward_implications":["One-shot motion customization is feasible: a single reference video can transfer a complex motion like skateboarding or playing guitar to an arbitrary new subject.","Few-shot customization from about 4–10 reference videos further improves text alignment, entity alignment, and temporal consistency, and the method also supports simultaneous subject and motion customization from an exemplar image set.","The decoupling is measurable: removing the recaptioner or the appearance injector lowers CLIP text/entity alignment while raising motion fidelity, consistent with overfitting to appearance and motion.","Training fits on a single A100 GPU with roughly 600 steps, and inference produces 24-frame clips at 576×320 in about 19 seconds, so the pipeline is lightweight for short-clip production.","The proposed Motion Fidelity (MoFid) metric, built on VideoMAE embeddings, gives a quantitative handle on whether generated videos actually perform the reference motion, complementing CLIP-based metrics."],"supporting_citations":[{"why":"The base text-to-video model (ZeroScope) that MoTrans fine-tunes; all experiments and comparisons are built on it.","marker":"[5]"},{"why":"DreamVideo, the prior work whose appearance-injection idea MoTrans adapts and extends; it is also a comparison baseline.","marker":"[45]"},{"why":"MotionDirector, the main baseline for dual-branch motion customization; provides the comparison that MoTrans claims to outperform.","marker":"[55]"},{"why":"Tune-A-Video, a one-shot T2I-based customization baseline used to show the limits of single-video transfer.","marker":"[46]"},{"why":"LAMP, a few-shot motion pattern learning baseline used in the few-shot comparison.","marker":"[47]"},{"why":"VideoMAE, the video encoder used to define the Motion Fidelity (MoFid) metric for evaluating motion similarity.","marker":"[41]"},{"why":"OpenCLIP ViT-H/14, the image encoder used for appearance injection and for CLIP-based metrics.","marker":"[10]"},{"why":"The encoder-based domain tuning work that supplies the L2 regularization on residual embeddings to keep them stable.","marker":"[13]"}],"fun_headline_variants":["Decouple appearance and motion for one-shot video transfer","One video, any subject: motion transfer via appearance-motion split","MoTrans: transfer motion from a single reference clip","Separate look and movement to transfer motion from one video","Two-stage LoRA: motion learned apart from appearance for transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The decoupling claim rests on the assumption that broadcasting a randomly chosen frame's image embedding into the UNet hidden states just before the temporal transformers steers the temporal LoRAs to learn motion only, rather than leaking the reference video's appearance into the generated video.","fun_headline_variants_meta":{"raw":{"variants":["Decouple appearance and motion for one-shot video transfer","One video, any subject: motion transfer via appearance-motion split","MoTrans: transfer motion from a single reference clip","Separate look and movement to transfer motion from one video","Two-stage LoRA: motion learned apart from appearance for transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1583,"prompt_tokens":934,"completion_tokens":649,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":567}},"tokens_in":550,"tokens_out":649,"duration_ms":6651,"temperature":1.0,"reasoning_tokens":567,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:26:36.924302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train MoTrans on two reference sets with the same motion but very different appearances (for example, a human lifting weights and a fluffy teddy bear lifting weights), generate videos of a new subject such as a tiger, and compare the outputs for appearance differences beyond sampling noise. If the tiger's texture or identity systematically changes with the reference set, the appearance injector is leaking appearance and the decoupling claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The base text-to-video model (ZeroScope) that MoTrans fine-tunes; all experiments and comparisons are built on it."},{"cited_title":"DreamVideo: Composing Your Dream Videos with Customized Subject and Motion","cited_arxiv_id":"2312.04433","evidence_quote":"DreamVideo, the prior work whose appearance-injection idea MoTrans adapts and extends; it is also a comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The encoder-based domain tuning work that supplies the L2 regularization on residual embeddings to keep them stable."}],"review_version":1}