{"id":"eb7f52b1-096f-48de-8960-fa344205ea7b","arxiv_id":"2504.14977","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Simple conditioning patches and training tricks on the Wan-2.1 model outperform specialized reference-network methods for controllable character animation, according to the paper's benchmarks.","lead":"RealisDance-DiT animates a character from a single photo using a pose video, built by making small changes to the Wan-2.1 video generation model. The paper argues that with a powerful foundation model, simple conditioning inputs outperform complex reference networks, and it adds a new benchmark for open-world character animation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central architectural claim is not controlled: the \"Reference Net inferiority\" test uses an under-trained pruned variant, and all headline comparisons confound the 14B Wan base with the proposed modifications.","rationale":"The reader's verdict of CONDITIONAL is appropriate, and the underlying concerns about unmatched baselines and the pruned Reference Net are already present in the reader's rationale. However, the reader's designated weakest assumption is the reliance on external pose estimators. In my reading, that dependency is a genuine but secondary limitation: it is explicitly acknowledged in Section 5, it affects essentially all pose-driven character animation methods, and the paper's core architectural argument does not hinge on it. The more load-bearing weakness is that the paper's central conceptual claim—Reference Net designs are suboptimal for large DiT models and simple modifications suffice—is not supported by a controlled comparison. Table 4 compares against a pruned Reference Net that never trains at full scale, and the headline comparisons against other methods confound the architecture with the Wan-2.1 14B base model. These two issues together mean that even if the proposed system is a strong baseline, the paper has not established why it is strong. The concrete test I propose, a matched-scale comparison on Wan-2.1 T2V 1.3B where the full Reference Net fits, would directly settle whether the simple modification is the causal factor. Since the empirical baseline itself remains plausible and the paper's own framing is already modest in the limitations section, the conditional verdict stands; no change to the reader's judgment is required.","tokens_in":11393,"tokens_out":5744,"duration_ms":56763,"concrete_test":"Run a controlled matched-scale ablation on a DiT-based foundation model small enough to fit the full Reference Net, for example Wan-2.1 T2V 1.3B, which the authors already use in other ablations. Fine-tune two variants with identical data, optimizer, learning rate, batch configuration, and total iterations: (a) a full Reference Net variant with the standard design, and (b) the proposed concatenated-reference latent with shifted RoPE. Evaluate both on the same subset of RealisDance-Val and report FVD and FID over at least three seeds with error bars. If variant (a) matches or beats variant (b), the conclusion that Reference Nets are suboptimal for DiTs fails, and the RealisDance-DiT result becomes re-interpretable as a base-model-scale effect; if variant (b) wins on both metrics, the architectural claim is supported at a scale where the full Reference Net is actually trainable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's conceptual claim is that for powerful DiT video foundation models, simple modifications are enough and the widely adopted Reference Net is suboptimal. The load-bearing evidence is Table 4, which compares full fine-tuning (FID 25.58, FVD 519.22), part fine-tuning (FID 24.79, FVD 563.28), and a \"Light Ref. Net\" variant (FID 31.01, FVD 678.98). But the full Reference Net could not be trained at all due to GPU memory (marked OOM); the compared Reference Net was pruned to one block per five main-network blocks. No convergence criterion, training steps, or final loss are reported for that variant, so its failure may simply reflect an under-trained, capacity-limited network rather than a fundamental design flaw. The paper also claims part fine-tuning \"will not degrade the final performance,\" yet Part Ft improves FID by 0.79 but worsens FVD by 44.06 relative to Full Ft, a discrepancy not discussed. Separately, every headline comparison against existing methods (Tables 1-3) uses RealisDance-DiT built on Wan-2.1 I2V 14B, while Animate-X, ControlNeXt, MimicMotion, MusePose, and MooreAA are built on far smaller Stable Diffusion UNet backbones. The \"large margin\" FVD/FID gains, and much of the qualitative superiority, could therefore be attributable to base-model scale and pre-training rather than the specific architectural recipe. The acknowledged pose-estimation limitation in Section 5 is real, but it is common to all pose-conditioned methods and is not the main threat to the central argument; the uncontrolled architecture comparison is. The paper may still deliver a strong empirical baseline, but its causal claim that Reference Nets are suboptimal for large DiTs is underdetermined by the evidence presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RealisDance-DiT, a controllable character animation method built on the Wan-2.1 I2V 14B video foundation model. The core claim is that with a sufficiently powerful foundation model, simple architectural modifications (concatenation of reference and pose latents, shifted RoPE) and two fine-tuning strategies (low-noise warmup timestep sampling and 'large batches, small iterations') can replace the heavy Reference Net designs used in prior methods. The authors introduce a new test set, RealisDance-Val, and report FVD/FID improvements over Animate-X, ControlNeXt, MimicMotion, MooreAA, and MusePose on TikTok, UBC Fashion, and RealisDance-Val. The paper also contains ablations on architecture and fine-tuning strategies, and a limitations section.","tokens_in":11633,"tokens_out":5642,"duration_ms":46991,"significance":"If the central claim is correct, this is a useful counterpoint to the prevailing Reference Net design for large DiT video models, and the proposed baseline plus the new benchmark would be a service to the community. The paper is clearly written and the qualitative results are visually compelling. Credit is due for explicitly acknowledging the dependency on three external pose estimators and for stating the 'large batches, small iterations' recipe. However, the empirical support for the headline claims is currently confounded by base-model scale and by an under-documented Reference Net ablation, so the significance can only be provisionally assessed.","major_comments":[{"comment":"The headline 'large margin' comparisons are confounded by base-model scale: RealisDance-DiT is built on Wan-2.1 I2V 14B, while Animate-X, ControlNeXt, MimicMotion, MooreAA, and MusePose are based on substantially smaller Stable Diffusion UNet backbones. The reported FVD/FID gains may therefore reflect the pre-training and capacity of the 14B base rather than the proposed architectural modifications or fine-tuning strategies. Please provide a matched-scale baseline, e.g., fine-tune Wan-2.1 I2V 14B with the same data but with a standard Reference Net or ControlNet-style injection, or at least report the performance of the unmodified Wan-2.1 I2V with pose conditioning via existing adapters, to isolate the contribution of the proposed recipe.","section":"§4.1, Tables 1-3"},{"comment":"The conclusion that 'Reference Net is suboptimal for large DiT models' is not supported by the presented evidence. The full Reference Net variant could not be trained (OOM), and the 'Light Ref. Net' variant prunes the Reference Net to one block per five main-network blocks with no reported convergence criterion, number of training steps, or final loss. Its inferior FID (31.01) and FVD (678.98) may be due to under-training or capacity reduction rather than the Reference Net design itself. Please train a Reference Net variant at matched scale and compute budget (e.g., with gradient checkpointing, offloading, or a lighter full Reference Net) and report training curves and final loss. Also, the claim that part fine-tuning 'will not degrade the final performance' is internally contradicted: Part Ft improves FID (24.79 vs 25.58) but worsens FVD (563.28 vs 519.22); please discuss this trade-off or provide repeated runs with confidence intervals.","section":"§4.2, Table 4"},{"comment":"The low-noise warmup strategy depends on hyperparameters α and τ, but the paper does not report the values used for the final model, nor any sensitivity analysis. The ablation in Figure 8 is on the 1.3B T2V model and only shows smoothed loss curves, not the effect on FVD/FID or generation quality. Please quantify the impact of the proposed sampling schedule on the final metrics (e.g., uniform vs. low-noise warmup on the same base model) and report α, τ, and the total number of training iterations, along with the batch sizes used in the 'large batches and small iterations' strategy.","section":"§3.2, Eqs. (2)-(3)"},{"comment":"The paper acknowledges that when all three pose estimators fail, the model generates random poses and artifacts. This limitation is common to pose-conditioned methods, but it interacts with the central claim: the evaluation and the method's success depend on upstream pose quality. The paper should quantify how often pose estimates are inaccurate on the RealisDance-Val dataset and discuss the impact on the reported metrics, or at least include a robustness analysis to noisy pose inputs.","section":"§5 (Limitations)"}],"minor_comments":[{"comment":"The 'zero projection layer' is mentioned in the overview but never formally defined; please specify its role, placement, and initialization.","section":"§3.1"},{"comment":"There is a typo 'RealiDance-DiT' in the qualitative comparison paragraph; it should be 'RealisDance-DiT'.","section":"§4.1"},{"comment":"The text describes different batch configurations but does not state the actual batch sizes and iteration counts used to produce Figure 9; please provide these details.","section":"§4.2, Figure 9"},{"comment":"On the UBC Fashion dataset, RealisDance-DiT ranks second on FVD and FID behind Animate-X; the phrase 'large margin' in the abstract and conclusion should be qualified to reflect that the large gains appear only on the proposed RealisDance-Val benchmark.","section":"§4.1, Table 3"},{"comment":"The RealisDance-Val dataset description (100 videos) lacks basic statistics such as resolution distribution, number of characters, scene types, and pose complexity measures; please include a summary table or histogram.","section":"§4 (Dataset)"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is plausible but the current evidence does not rule out the alternative explanation that the gains come from the Wan-2.1 base model. I would encourage the editor to require the matched-scale control experiment before acceptance. Also, given the use of vendor-purchased training data and the lack of code/weights at review time, the authors should clarify data licenses and release code, model weights, and the RealisDance-Val dataset to make the strong claims verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth a look, but the headline claim should be treated with caution. It argues that for a powerful video DiT like Wan-2.1, you do not need a Reference Net; simple conditioning patches and fine-tuning tricks suffice. That is an interesting and actionable idea. The recipe is genuinely simple: concatenate pose latents, use a shifted RoPE, and fine-tune the 14B model with a low-noise warmup schedule and large batches with few iterations. The authors also contribute a new challenging test set, RealisDance-Val, with 100 open-scene videos.\n\nWhat the paper does well: The internal ablations (Table 4) show that simple full or partial fine-tuning beats a pruned Reference Net variant on FID/FVD, and that part fine-tuning lands close to full fine-tuning. The low-noise warmup strategy is clearly explained and the loss curves support faster convergence. The writing is honest about limitations: Section 5 states that when all three pose estimators fail, the model produces random poses and artifacts, and that static-background failure mode is documented.\n\nNow the soft spots, in proportion. The biggest one: the central causal claim is not controlled. The full Reference Net OOM, so the comparison is against a version with one block per five main blocks, and no convergence check or training budget is reported for that variant. That could be an under-trained capacity-limited model, not a fundamental design flaw. Second, every headline comparison against Animate-X, ControlNeXt, MimicMotion, MusePose, and MooreAA is a 14B DiT versus much smaller UNet models. The \"large margin\" in FVD/FID is very plausibly the base model's scale and pre-training, not the specific recipe. There are no error bars, and the new benchmark is self-curated with manual reference frame selection, which invites subtle bias. The pose-estimation dependency is real but common to all pose-driven methods; it is a limitation, not a unique flaw.\n\nNet assessment: the paper is a useful empirical baseline and the architectural simplicity is refreshing. But the evidence does not support the categorical claim that Reference Nets are suboptimal for large DiTs. A serious referee could ask for a matched-scale control (e.g., a standard Reference Net on a similar-size DiT, or fine-tuning a smaller Wan model) and for error bars. I would accept it for peer review, but major revision is needed.\n\nRecommended for a reading group if you are interested in how base-model scale confounds architecture comparisons.","headline":"A plausible simple-recipe baseline for controllable character animation on a large DiT, but the headline causal claim about Reference Nets is not actually controlled.","tokens_in":12313,"tokens_out":2453,"would_cite":true,"duration_ms":21832,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a powerful video foundation model can be adapted to controllable character animation with minimal architectural edits, outperforming existing methods built on heavy reference networks.","keywords":["controllable character animation","video diffusion model","DiT foundation model","Reference Net","low-noise warmup","large batches small iterations","pose-guided generation","RealisDance-Val"],"falsifier":"Evaluate the model on a set of extreme, heavily occluded poses (for example acrobatic or gymnastic sequences) where HaMeR, DWPose, and SMPL-CS are known to be inaccurate, and measure generated frames' pose alignment and artifact rate. If outputs remain clean and pose-accurate, the claim that simple modifications suffice in the wild is supported; if random poses and artifacts appear, the boundary described in the paper's limitation section is reached.","tokens_in":11104,"feed_emoji":"🎬","tokens_out":8946,"duration_ms":70894,"temperature":0.7,"pith_summary":"RealisDance-DiT asks what happens when you stop adding heavy reference networks to a video diffusion model and start from a much larger pretrained video foundation model. The paper's answer is that a few simple changes — concatenating the pose and reference conditions into the DiT's latents, using a spatially shifted rotary position embedding for the reference latent, and fine-tuning only a subset of parameters — are enough for state-of-the-art controllable character animation. To keep the foundation model's priors intact, it proposes two fine-tuning tactics: a low-noise warmup that samples small diffusion timesteps early, and a \"large batches, small iterations\" schedule that limits the number of adaptation steps. Evaluated on a new open-world benchmark (RealisDance-Val) plus the TikTok and UBC fashion video datasets, the model reports the best or second-best score on most metrics, with the largest margins on FVD and FID. If the claim is right, the field should spend less effort on bypass networks and more on scaling and preserving strong video foundation models.","feed_headline":"Simple edits to a large video model beat heavy animation networks","feed_subtitle":"A large pretrained video diffusion model can animate characters in the wild with just a few conditioning edits.","key_machinery":"The load-bearing mechanism is the minimal conditioning path into the DiT: pose latents from HaMeR, DWPose, and SMPL-CS are encoded by the Wan VAE and added to the noise latent, while the reference latent is concatenated along the sequence dimension, with its rotary position embedding spatially shifted to the first frame. This replaces the Reference Net with a small set of trainable components — condition patchifiers, a zero projection layer, and the self-attention blocks — so the foundation model's own priors do the work. The second mechanism is the fine-tuning recipe: the low-noise warmup defines a dynamic probability density $f(x)$ that samples small timesteps early and decays to uniform sampling after $\\tau$ iterations, and \"large batches, small iterations\" keeps updates informative while limiting the number of adaptation steps to protect pretrained priors.","core_discovery":"The central discovery is that the widely adopted Reference Net — a separate branch that injects character identity into the generative model — is suboptimal for large DiT-based video foundation models like Wan-2.1. Minimal modifications, namely adding pose latents to the noise latent, concatenating the reference latent along the sequence dimension, and replacing the rotary position embedding with a spatially shifted version so the reference latent sits at the first frame, produce a stronger baseline than the Reference Net variants tested. The paper argues this works because the foundation model already contains the priors needed for open-world character consistency; fine-tuning's job is to unlock those priors, not to add structure. Two training strategies support this: a dynamic timestep-sampling distribution that favors low-noise samples early in fine-tuning, and a \"large batches, small iterations\" schedule that prevents downstream data from overwriting the pretrained prior. The resulting RealisDance-DiT is reported to outperform all compared methods by a large margin on FVD and FID on RealisDance-Val and to rank first or second across metrics on TikTok and UBC fashion video.","pith_inferences":["If RealisDance-DiT's success generalizes, future gains in character animation may come mainly from scaling and improving the video foundation model itself, rather than from inventing more elaborate conditioning architectures.","The method's dependence on HaMeR, DWPose, and SMPL-CS means its \"in the wild\" scope is bounded by pose-estimation quality; training with corrupted or noisy poses, or with a learned robust pose encoder, could push that boundary further than the paper's limitation section acknowledges.","The \"large batches, small iterations\" recipe is plausibly a general fine-tuning principle for other large generative models on narrow downstream tasks, where preserving pretrained priors matters more than minimizing the training loss.","The static-background failure in stationary-camera scenes suggests the model lacks explicit camera-motion or optical-flow conditioning; adding such cues could be a next step for cases where the pose sequence carries little motion information."],"forward_implications":["Reference Net branches become unnecessary for large DiT video foundation models; the concatenation-plus-shifted-RoPE design matches or beats them while adding far fewer parameters.","Fine-tuning only a subset of parameters (condition patchifiers, zero projection layer, self-attention blocks) is enough for downstream character animation, so the foundation model's priors are largely preserved.","Sampling low-noise examples early in fine-tuning accelerates convergence, while a high-noise warmup slows it down, indicating that early adaptation should avoid difficult samples.","Training with large batches and few iterations prevents overfitting to downstream data; too many iterations reduce output diversity and introduce background artifacts.","The new RealisDance-Val benchmark exposes open-world challenges — rare poses, stylized characters, object interactions, and complex lighting — that simpler benchmarks do not capture, and the proposed method performs best on the aggregate FVD and FID."],"supporting_citations":[{"why":"Wan-2.1, the large DiT video foundation model that supplies the pretrained priors and backbone for RealisDance-DiT.","marker":"[29]"},{"why":"Animate Anyone, the Reference Net-based approach the paper contrasts with and argues is suboptimal for large DiT models.","marker":"[9]"},{"why":"RealisDance, the prior work whose three pose conditions (HaMeR, DWPose, SMPL-CS) and hand-focused setting this method inherits.","marker":"[35]"},{"why":"HaMeR, the 3D hand pose estimator that provides one of the three pose condition inputs.","marker":"[17]"},{"why":"DWPose, the whole-body pose estimator providing the body condition.","marker":"[31]"},{"why":"VBench-I2V, the evaluation metric used on the RealisDance-Val benchmark for open-scene comparisons.","marker":"[11]"},{"why":"HumanVid, whose evaluation protocol on TikTok and UBC fashion video datasets the paper follows.","marker":"[28]"}],"fun_headline_variants":["Minimal edits to Wan-2.1 outdo heavy reference nets","Simple DiT tweaks beat complex animation networks","RealisDance: Less architecture, more priors","Drop Reference Net: Small DiT changes win","Open-world character animation just got simpler"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the accuracy of three external pose estimators; when all three misestimate an extremely complex pose, the model cannot compensate and tends to generate random poses with artifacts, as the paper's own limitation section concedes.","fun_headline_variants_meta":{"raw":{"variants":["Minimal edits to Wan-2.1 outdo heavy reference nets","Simple DiT tweaks beat complex animation networks","RealisDance: Less architecture, more priors","Drop Reference Net: Small DiT changes win","Open-world character animation just got simpler"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1306,"prompt_tokens":1010,"completion_tokens":296,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":220}},"tokens_in":626,"tokens_out":296,"duration_ms":3238,"temperature":1.0,"reasoning_tokens":220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:35:33.967748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the model on a set of extreme, heavily occluded poses (for example acrobatic or gymnastic sequences) where HaMeR, DWPose, and SMPL-CS are known to be inaccurate, and measure generated frames' pose alignment and artifact rate. If outputs remain clean and pose-accurate, the claim that simple modifications suffice in the wild is supported; if random poses and artifacts appear, the boundary described in the paper's limitation section is reached.","supporting_citations":[{"cited_title":"Animate anyone: Con- sistent and controllable image-to-video synthesis for character animation","cited_arxiv_id":null,"evidence_quote":"Animate Anyone, the Reference Net-based approach the paper contrasts with and argues is suboptimal for large DiT models."},{"cited_title":"Reconstructing hands in 3d with transformers","cited_arxiv_id":null,"evidence_quote":"HaMeR, the 3D hand pose estimator that provides one of the three pose condition inputs."},{"cited_title":"Effectivewhole-bodyposeestimationwith two-stages distillation","cited_arxiv_id":null,"evidence_quote":"DWPose, the whole-body pose estimator providing the body condition."},{"cited_title":"Humanvid: Demystifying training data for camera-controllable human image animation","cited_arxiv_id":null,"evidence_quote":"HumanVid, whose evaluation protocol on TikTok and UBC fashion video datasets the paper follows."}],"review_version":1}