{"id":"d3fdc7fd-a76b-4540-b36e-ad7e145329cb","arxiv_id":"2412.17290","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-reference diffusion network with pose-correlated adaptive reference selection improves free-viewpoint human animation under large viewpoint and camera-distance changes.","lead":"This paper introduces a diffusion-based method that animates a human character from multiple reference photos, using pose similarities to pick the most useful regions when the camera viewpoint changes dramatically. It could make it easier to generate cinematic shot changes, like zooming from a full body to a close-up, without losing the character's identity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA comparison in Tables 2-3 lacks same-training-data baselines; reported gains may reflect MSTed training exposure rather than pose-correlated reference selection.","rationale":"Reading in good faith, the paper proposes a coherent multi-reference architecture with a pose-correlation guided selection mechanism, and its internal ablations do suggest each component contributes. The central claim, however, is an empirical superiority claim, and its soundness depends on whether the comparison against SOTA methods is apples-to-apples. The paper reports only that Ours(R=1) uses one reference at inference to 'match the settings of one-reference approaches', but it does not state whether the baselines were retrained on the same data. Since MSTed is introduced by this paper and is specifically designed to be more challenging for viewpoint and camera-distance changes, released single-reference checkpoints are likely at a domain disadvantage. The internal baseline in Table 4 is the correct control, but it is omitted from the SOTA tables. The pose-correlation assumption identified by the reader is a genuine secondary risk: if the correlation maps are not meaningful under non-aligned poses, the top-K selection could become near-random. Yet even that failure would not by itself invalidate the central empirical claim if multi-reference training still helps; by contrast, an unfair baseline comparison directly undermines the claim that the model is better than SOTA. I therefore agree with the CONDITIONAL verdict, but the most load-bearing condition is a same-data retraining protocol for the baselines, plus release of code and data to make the comparison reproducible.","tokens_in":16444,"tokens_out":8743,"duration_ms":92042,"concrete_test":"Retrain AnimateAnyone and Champ on the exact MSTed training split used for Ours (same resolution, pose representation, and R=1 inference), evaluate on the same test identities, and report the same metrics. Then compare against Ours(R=1) and the Table 4 single-reference baseline. If a retrained baseline closes the LPIPS/FVD gap by more than half, the improvement is attributable to training data or protocol rather than to pose-correlated reference selection; if it does not, the SOTA claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract's central claim is that, with the same number of reference images, the model outperforms current SOTA under large viewpoint change. The evidence for this is Tables 2-3, but Section 4.2 never specifies whether MagicAnimate, AnimateAnyone, and Champ were retrained on the MSTed/DyMVHumans training splits or evaluated with released checkpoints. Because MSTed is a new, deliberately harder domain with large camera-distance changes, a pretrained single-reference model is at an unknown but substantial disadvantage; the reported gaps (e.g., Table 2 LPIPS 0.2045 vs 0.4618, FVD 20.88 vs 25.68) are large enough that training-data exposure rather than the pose-correlation selection mechanism could explain them. The only same-data control is the 'baseline' in Table 4, which is not shown alongside the SOTA rows. Without a same-training-protocol comparison, the headline superiority claim is confounded by training data and evaluation protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a diffusion-based human animation method that accepts multiple reference images to handle large viewpoint and camera-distance changes. A pose correlation module computes correlation maps between target and reference poses, and an adaptive reference selection strategy feeds only the top-K reference features into the denoising UNet to bound computational cost. The authors introduce MSTed, a new multi-shot TED-talk dataset of 1,084 identities with large viewpoint/distance variation, and evaluate on MSTed and DyMVHumans against MagicAnimate, AnimateAnyone, and Champ. The main claim is that, with the same number of reference images, the proposed model outperforms these SOTA methods under large viewpoint change, and that increasing the number of references further improves quality. The paper also includes ablations showing the contributions of the multiple-reference mechanism, pose correlation module, and reference selection.","tokens_in":16625,"tokens_out":4244,"duration_ms":40726,"significance":"If the reported results hold, the paper makes a useful contribution: it identifies a real limitation of single-reference human animation under viewpoint/camera-distance changes and proposes a practical multi-reference mechanism with a selection strategy that avoids unbounded growth in the denoising network's input. The MSTed dataset is a potentially valuable resource for training and benchmarking free-viewpoint human animation, with real-world zoom-in/zoom-out variation that existing multi-view studio datasets lack. The paper also provides quantitative tables, qualitative comparisons, and ablations. However, the central superiority claim depends on an evaluation protocol that is not fully specified, and no uncertainty estimates are given, so the strength of the evidence is currently limited.","major_comments":[{"comment":"The manuscript does not state whether MagicAnimate, AnimateAnyone, and Champ were retrained on the MSTed and DyMVHumans training splits or evaluated using their released checkpoints. Because MSTed contains substantially larger viewpoint and camera-distance variation than the typical training data of these single-reference methods, a released checkpoint may be at a large, unquantified disadvantage due to domain shift. The abstract's claim that \"with the same number of reference images, our model performs favorably\" is therefore confounded by training-data exposure unless the baselines are trained under the same protocol. Please specify the evaluation protocol explicitly; ideally, retrain or fine-tune all baselines on the same training data and report those numbers alongside the current ones.","section":"Section 4.2, Tables 2-3"},{"comment":"No error bars, confidence intervals, or multiple-seed runs are reported for any metric. While some gaps (e.g., Table 2 LPIPS 0.2045 vs. 0.4618) are large, other comparisons (e.g., Table 3 FVD Ours(R=1) 9.047 vs. AnimateAnyone 12.300; Ours(R=2) PSNR 32.49 vs. Ours(R=1) 32.18) are smaller and could lie within run-to-run variability. Please report mean and standard deviation over at least three random seeds, or bootstrap confidence intervals, and state the number of test clips used. This is necessary to assess whether the reported improvements are meaningful.","section":"Tables 2-4"},{"comment":"The central mechanism of pose-correlated reference selection is validated only through a single qualitative visualization (Figure 6) plus an ablation (Table 4). The paper would be substantially strengthened by a quantitative evaluation of the selection mechanism itself, for example: (i) compare the selected top-K tokens against ground-truth visible body-part annotations on a subset of data; (ii) sweep K and show the effect on quality and computational cost; or (iii) compare against alternative selection strategies (random selection, uniform attention, or full-feature concatenation without selection). As written, the reader cannot tell whether the correlation map actually localizes informative regions or merely acts as a learned soft gate with no specific pose-correspondence semantics.","section":"Section 3.3 and Figure 6"}],"minor_comments":[{"comment":"The abstract says the method \"eliminate[s] the computational cost\" of multiple references, but the reference UNet still encodes all N reference images and the pose correlation module computes N×T correlation maps. Only the denoising UNet's input is bounded by top-K selection. Please rephrase to \"reduces the marginal cost in the denoising UNet\" to avoid overclaiming.","section":"Abstract and Section 3.3"},{"comment":"The caption of Table 4 does not state which dataset the ablation is performed on; please add this information, as the conclusion \"Ours (R=2) achieves the best results\" may not transfer to DyMVHumans.","section":"Section 4.3, Table 4"},{"comment":"Training details are minimal: the paper only says \"We follow the training manner as in AnimateAnyone and Champ.\" Please provide key hyperparameters, the value of M (maximum reference count during training), the value of K_l per layer, the number of GPUs, training iterations, and the resolution used, so that the experiments can be reproduced.","section":"Section 3.4"},{"comment":"The caption contains a typo: \"Ground True\" should be \"Ground Truth.\"","section":"Figure 4"},{"comment":"There is a typo: \"keypoitns pairs\" should be \"keypoint pairs.\" Also, some references (e.g., Champ) are cited in a preprint form without publication venue; please update if a published version exists.","section":"Section 2 (Related Work)"}],"recommendation":"major_revision","confidential_remarks":"The core idea is reasonable and the dataset is a useful contribution, but the evaluation protocol issue (whether SOTA baselines were retrained on MSTed/DyMVHumans) must be resolved before the headline claim can be accepted. I would suggest the editor require the authors to either clearly state that released checkpoints were used and discuss the domain-shift caveat, or—preferably—retrain/fine-tune the baselines on the same training splits and report those numbers. Additionally, the lack of error bars is a recurring problem in generative-model papers; asking for multiple seeds or confidence intervals would improve the reliability of the comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's new: the paper extends the double-UNet recipe to multiple reference images, selects the most useful reference tokens via pose-correlated attention, and contributes MSTed, a 1,084-identity multi-shot TED dataset. That combination is genuinely absent from the cited single-reference methods. The pose correlation module and the top-K selection with random sampling during training to work around the non-differentiable argsort are both sound engineering. The ablation table shows each piece adds something: two references help, the correlation module helps, the selection keeps quality while cutting compute. I buy the architecture.\n\nThe soft spots are real but fixable. Section 4.2 compares against MagicAnimate, AnimateAnyone, and Champ without saying whether those baselines were retrained on MSTed/DyMVHumans or run with official checkpoints. The domain shift from a pretrained checkpoint to a new, harder dataset with zoom-in/zoom-out is a serious confound, and it directly undercuts the abstract's claim that 'with the same number of reference images' the method beats SOTA: your R=1 model was trained with multiple references, so it is not actually the same training setting. No error bars are reported anywhere, and the only evidence for the correlation map's behavior is Figure 6. These are the kind of things a reviewer should catch, and they are probably fixable without changing the method.\n\nThe dataset deserves credit. The construction is transparent — DINOv2 for shot segmentation, YOLO for single-person filtering — and it fills a real gap, since MVHuman and DyMVHumans don't vary camera distance. The limitation section honestly notes the domain is mostly talk-style video.\n\nI don't think the central argument collapses. The stress-test worry about training data exposure is fair, but the ablation gives a same-data control that partially covers it, and the learned components are coherent. The paper is worth sending to a serious referee; it should not be desk rejected. I'd make the decision conditional on the authors clarifying baseline training protocols, adding error bars, and ideally releasing code and data. It's a solid within-subfield contribution, not a breakthrough.","headline":"A solid multi-reference human animation paper with a real dataset and a coherent selection mechanism, but the headline SOTA claim rests on an uncontrolled baseline comparison.","tokens_in":17163,"tokens_out":2573,"would_cite":true,"duration_ms":25876,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pose-correlated reference selection network uses multiple reference images to animate humans under large viewpoint changes.","keywords":["human animation","free-viewpoint video synthesis","diffusion models","multiple reference images","pose correlation","adaptive reference selection","novel view synthesis","character animation"],"falsifier":"Take the trained model and replace the top-K correlation selection at test time with a random selection of the same number of reference feature tokens. If L1, PSNR, LPIPS, MOVIE, and FVD on the large-viewpoint test splits of both datasets stay essentially unchanged, then pose-correlated selection is not what drives the quality gains, and the evidence would shift to the multiple-reference training alone. A second check is to inspect whether the top-K tokens' spatial locations match the reference regions a human would call informative, such as head, hands, and visible torso, across clips with zoom-in and zoom-out; the current qualitative support is a single visualization.","tokens_in":16229,"feed_emoji":"🎥","tokens_out":6045,"duration_ms":49809,"temperature":0.7,"pith_summary":"The paper tries to establish that reference-based human animation can survive large viewpoint and camera-distance changes if the network is allowed to look at several reference images instead of one, and if it is told which parts of those references are relevant for the target pose. The authors argue that a single image simply lacks the appearance information needed for close-up and wide-shot variations, so they add a pose correlation module that compares target and reference poses and an adaptive selection that keeps only the most relevant reference features. On a new multi-shot dataset of public talks and on the DyMVHumans benchmark, they report that their model with the same number of references outperforms current single-reference methods under large viewpoint change, and that quality improves as more references are added. If true, this would make camera control and shot-type planning possible for image-driven human animation without relying on 3D reconstruction.","feed_headline":"Chosen references let animated humans survive big viewpoint shifts","feed_subtitle":"Pose-correlated selection finds the useful parts of extra reference photos, keeping identity stable in close-ups and wide shots.","key_machinery":"The load-bearing mechanism is the pose correlation module paired with adaptive reference selection. For each reference pose $P^i_{\\text{ref}}$ and target pose $P^j_{\\text{tgt}}$, two separate pose encoders produce features $F^i_{\\text{ref}}$ and $F^j_{\\text{tgt}}$; a transformer with cross-attention then builds a correlation map $R_{i,j} = f_{\\text{zero}} \\circ T(W_q F^i_{\\text{ref}}, W_k F^j_{\\text{tgt}}, W_v F^j_{\\text{tgt}})$ that scores how informative each spatial region of the reference is for the target viewpoint. These scores are flattened and concatenated, and the top-$K_l$ reference tokens are selected and multiplied by their correlation values before being injected into each denoising UNet layer. A uniform-sampling companion term during training keeps the non-differentiable argsort from trapping the network in a poor local minimum.","core_discovery":"On the paper's own terms, the central discovery is that the bottleneck for free-viewpoint human animation is not the diffusion model's capacity but the information supplied by a single reference image. The authors show that a double-UNet diffusion architecture can be extended to accept $N$ reference images at roughly constant cost if a learned pose correlation module produces a correlation map $R_{i,j}$ between each reference pose and the target pose, and a top-$K$ selection step keeps only the most correlated reference feature tokens. They further show that training with multiple references improves results even when only one reference is available at inference, and that the correlation map visually concentrates on informative regions such as the head and hands. The quantitative claim is that, with the same number of reference images, this design performs favorably against current single-reference methods under large viewpoint change.","pith_inferences":["Beyond the paper, correlation-guided token pruning could generalize to any multi-view conditioning task where a cheap geometric signal such as pose, depth, or camera parameters stands in for dense appearance correspondence, not just human animation.","Beyond the paper, the design implies a testable separation: because top-K selection is only needed to control cost, one could ablate the correlation module against random selection under a fixed compute budget; if random tokens perform equally, the module's benefit would be attributable to the multiple-reference training rather than to pose-correlated selection.","Beyond the paper, since the correlation maps encode which reference body regions match a target viewpoint, they could be reused as soft correspondences for camera or viewpoint estimation, a use the paper does not explore."],"forward_implications":["If the reported comparisons hold, adding more reference images improves video quality under viewpoint change, with measurable gains from one to two references on both evaluated datasets.","If the reported comparisons hold, training on multiple references transfers to single-reference inference, so even users who provide one image get better appearance consistency than single-reference baselines.","If the reported comparisons hold, the top-$K$ selection keeps inference cost roughly independent of the number of references, allowing up to ten reference images in the reported experiments.","If the reported comparisons hold, the new multi-shot dataset supplies an in-the-wild benchmark with both viewpoint and camera-distance variation, which the authors say no previous public dataset provided."],"supporting_citations":[{"why":"It supplies the double-UNet backbone and the single-reference baseline that the method extends.","marker":"[9]"},{"why":"It is the strongest single-reference baseline with multi-modal driving signals against which the method compares.","marker":"[40]"},{"why":"It provides an additional single-reference diffusion baseline used in the viewpoint-change comparisons.","marker":"[34]"},{"why":"It provides the DyMVHumans multi-view benchmark used to validate generalization across viewpoints.","marker":"[38]"},{"why":"It is the self-supervised visual feature model used to segment public talk videos into single-shot clips for the MSTed dataset.","marker":"[17]"},{"why":"It is the object detector used to filter clips for consistent single-person content in the MSTed dataset.","marker":"[19]"}],"fun_headline_variants":["Pose-correlated refs stabilize human animation across views","Adaptive reference selection powers free-viewpoint human animation","Multiple references improve human animation under large view shifts","Correlated reference regions keep animated humans recognizable","Viewpoint-robust human animation via smart reference selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the pose correlation module's cross-attention producing meaningful similarity scores between target and reference pose features even when the poses are not spatially aligned; if those scores do not localize the informative reference regions under large viewpoint changes, the top-K selection would keep near-random tokens and the reported gains would disappear.","fun_headline_variants_meta":{"raw":{"variants":["Pose-correlated refs stabilize human animation across views","Adaptive reference selection powers free-viewpoint human animation","Multiple references improve human animation under large view shifts","Correlated reference regions keep animated humans recognizable","Viewpoint-robust human animation via smart reference selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1218,"prompt_tokens":943,"completion_tokens":275,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":200}},"tokens_in":559,"tokens_out":275,"duration_ms":3295,"temperature":1.0,"reasoning_tokens":200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:37:21.368926+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained model and replace the top-K correlation selection at test time with a random selection of the same number of reference feature tokens. If L1, PSNR, LPIPS, MOVIE, and FVD on the large-viewpoint test splits of both datasets stay essentially unchanged, then pose-correlated selection is not what drives the quality gains, and the evidence would shift to the multiple-reference training alone. A second check is to inspect whether the top-K tokens' spatial locations match the reference regions a human would call informative, such as head, hands, and visible torso, across clips with zoom-in and zoom-out; the current qualitative support is a single visualization.","supporting_citations":[{"cited_title":"Animate anyone: Consistent and controllable image- to-video synthesis for character animation","cited_arxiv_id":null,"evidence_quote":"It supplies the double-UNet backbone and the single-reference baseline that the method extends."},{"cited_title":"Magicanimate: Temporally consistent human image animation using diffusion model","cited_arxiv_id":null,"evidence_quote":"It provides an additional single-reference diffusion baseline used in the viewpoint-change comparisons."},{"cited_title":"Pku- dymvhumans: A multi-view video benchmark for high- fidelity dynamic human modeling","cited_arxiv_id":null,"evidence_quote":"It provides the DyMVHumans multi-view benchmark used to validate generalization across viewpoints."},{"cited_title":"You only look once: Unified, real-time object detection","cited_arxiv_id":null,"evidence_quote":"It is the object detector used to filter clips for consistent single-person content in the MSTed dataset."}],"review_version":1}