{"id":"f3644c8b-4234-4be8-a9f7-f436030f7683","arxiv_id":"2412.13185","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Move-in-2D generates human motion sequences from a scene image and a text prompt, using a new large-scale motion dataset, and the generated motion improves downstream human video synthesis.","lead":"This paper trains a diffusion model that takes a scene image and a text prompt and outputs a 3D human motion sequence that fits the scene. It also introduces a 300,000-video dataset of human motion with text and background scenes, targeting more controllable human video generation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported FID/accuracy discard global translation and orientation, so they do not test the scene-projection claim; the only direct scene-alignment evidence is a 20-sample VLM study.","rationale":"The reader anchored on pseudo-label noise in 4D-Humans reconstructions. That is real, but it affects all compared methods similarly and does not specifically invalidate the relative ranking. The metric mismatch is more direct: the paper's own evaluation strips the very quantities (global orientation and translation) that encode the 'project naturally onto the scene image' part of the claim. If the scene-conditioning mechanism works, its effect must show up in θg/π distributions and in 2D reprojection; Table 2 is blind to those. The VLM study is too small and uncalibrated to carry the burden alone. This is not a claim of fraud; it is an internal limitation of the evidence. A concrete fix — reporting projection-sensitive metrics on the full test set — would settle whether the central claim holds. The verdict remains CONDITIONAL: the method may well work, but the evidence as presented does not establish the scene-compatibility component. The reader's conditional verdict already anticipates this; our concern strengthens the reasons for requiring the additional analysis and code/data release, but does not move the verdict.","tokens_in":12558,"tokens_out":4428,"duration_ms":42599,"concrete_test":"Re-evaluate the full held-out test set (957 samples) with global orientation θg and translation π included, and compute a direct scene-projection error: project the predicted SMPL body with the predicted camera translation onto the input image and compare 2D keypoints to the 2D keypoints (e.g., OpenPose) from the corresponding original held-out video frames. Report per-joint 2D error, and also report FID/Accuracy on the full representation including θg and π for Ours, MDM+, and Ours-scene. If Ours does not outperform the image-agnostic MDM+ on this projection-sensitive metric, the scene-conditioning advantage claimed in Table 2 is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Abstract and Sec. 1) is that, given a scene image and text, the model generates a motion sequence that 'can be projected naturally onto the scene image.' Yet the quantitative evaluation in Sec. 5 explicitly removes the global parameters needed for that projection: 'To standardize outputs across models, we ignore global orientation and translation.' The motion representation is θb (23 joint rotations), global orientation θg, and camera translation π; Table 2's FID (44.639), Accuracy (0.661), Diversity, and Multimodality are all computed after discarding θg and π. These numbers therefore measure body-pose quality and action recognition, not whether the generated person is placed at a plausible scale and location in the scene. The only direct scene-alignment evaluation is the VLM study (Table 3), which averages over just 20 test videos, scores only the middle frame, and uses ChatGPT-4o with no reported inter-rater or repeat-run statistics. The claim that Ours is the first method to produce scene-compatible motion from 2D images thus rests on quantitative metrics that ignore the scene-compatibility coordinate, plus a small subjective study. This is a load-bearing gap: a model that ignores the image entirely (or predicts an average pose distribution) could still achieve good Table 2 scores, since those scores discard placement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Move-in-2D, a conditional diffusion model that generates SMPL motion sequences from a 2D scene image and a text prompt, with the stated goal of producing motion that 'can be projected naturally onto the scene image.' To support training, the authors introduce HiC-Motion, a dataset of 300k videos with pseudo-ground-truth SMPL motions extracted via 4D-Humans, inpainted background frames, and text captions. The model uses a transformer with in-context conditioning for text and scene tokens, AdaLN for the timestep, and classifier-free guidance. Experiments include FID, accuracy, diversity, multimodality, a VLM-based automated evaluation, and a downstream video-generation application with Champ and Gen-3. The central claim is that Move-in-2D is the first method to produce scene-compatible human motion from a 2D image plus text.","tokens_in":12834,"tokens_out":3894,"duration_ms":38024,"significance":"If the result holds, this is a valuable new task formulation and a large-scale dataset contribution. The HiC-Motion dataset at 300k sequences significantly exceeds existing scene-aware motion datasets, and the ablation study in Table 4 provides evidence that the in-context conditioning design contributes to the reported gains. The application to motion-guided video generation is practical and well-motivated. However, the evaluation does not currently substantiate the headline claim of scene compatibility: the main quantitative metrics discard the very global parameters needed for 2D projection, and the only direct scene-alignment evidence is a small VLM study. The reliance on pseudo-labels for both training and evaluation further weakens the empirical grounding. These issues are fixable, but they are load-bearing for the paper's central contribution.","major_comments":[{"comment":"The paper's central claim is that generated motion 'can be projected naturally onto the scene image,' but the main quantitative metrics are computed after explicitly ignoring global orientation and translation ('To standardize outputs across models, we ignore global orientation and translation'). FID, Accuracy, Diversity, and Multimodality therefore measure body-pose distribution quality and action recognition, not whether the person is placed at a plausible scale, location, or orientation in the scene. A model that ignores the image entirely could in principle score well on these metrics. The paper should add metrics that directly use the predicted global translation π and orientation θg, such as 2D reprojection error against the ground-truth projection, foot-contact or floor-penetration checks, or human evaluation of the rendered full sequence.","section":"Sec. 5, 'Evaluation metrics' and Table 2"},{"comment":"All metrics are reported as point estimates without variance, confidence intervals, or significance tests. The reported advantage of Ours over MDM+ (FID 44.639 vs. 46.035; Accuracy 0.661 vs. 0.620) may be within sampling noise, especially since both models are trained on the same data and the differences in Table 4 are similar in magnitude. Please report results over multiple random seeds or provide confidence intervals and, where appropriate, paired statistical tests.","section":"Table 2 and Table 4"},{"comment":"The only direct scene-alignment evaluation is the VLM study, which averages over just 20 test videos, scores only the middle frame of each sequence, and uses ChatGPT-4o without any reported inter-rater agreement, repeat-run variance, or confidence intervals. Twenty samples is far too small to support the claim that 'our method consistently outperforms' all baselines on scene alignment. The paper should evaluate on a substantially larger sample, score multiple frames per sequence, report the prompt template, and provide variance or agreement statistics.","section":"Sec. 5.2, 'Automated evaluation' and Table 3"},{"comment":"The ground-truth motions are pseudo-labels extracted by 4D-Humans, and the held-out test set is drawn from the same pseudo-labeled distribution. If these pseudo-labels contain errors—such as camera-motion contamination, drift, or failures under occlusion—those errors directly bias both training and evaluation. The paper should quantify the reliability of the pseudo-labels (e.g., by manually verifying a random subset or by comparing against a small motion-capture dataset), and should report how the optical-flow filtering in Sec. 4.4 changes the distribution of the pseudo-labels (e.g., the distribution of global translation magnitudes before and after filtering).","section":"Sec. 3, 'Data preprocessing', and Sec. 5, 'Evaluation data'"},{"comment":"The STGCN classifier used for FID and accuracy is trained by the authors on their own pseudo-labeled motion data. This makes the metrics sensitive to the classifier's idiosyncrasies and to the pseudo-label distribution, and it is not clear that the classifier generalizes to open-domain motions. Please report the classifier's accuracy on a held-out set, compare against a classifier trained on an independent motion dataset if possible, or provide an alternative human-evaluation metric for action alignment.","section":"Sec. 5, 'Evaluation metrics'"}],"minor_comments":[{"comment":"The term 'fixed-background videos' is not defined; please specify the optical-flow threshold used to select this subset, and state the median-flow criterion precisely so that the filtering step is reproducible.","section":"Sec. 4.4, 'Selection of fine-tuning set'"},{"comment":"The caption reads 'MDM and SceneDiff produces implausible poses'; the verb should agree with the plural subject ('produce').","section":"Sec. 5.1, Figure 5 caption"},{"comment":"The sentence 'prior works [19, 20, 46, 47, 57, 57]' cites reference [57] twice; one duplicate should be removed.","section":"Sec. 2, paragraph beginning 'Given a 3D indoor scene'"},{"comment":"The paper states that 'there is currently no established metric to assess the compatibility between generated motion sequences and 2D background images,' but the authors themselves use a VLM as a new metric; it would be helpful to state the exact prompt and scoring rubric used for ChatGPT-4o, and to note whether the evaluation was repeated.","section":"Sec. 5.2, 'Automated evaluation'"},{"comment":"The limitation 'our framework does not control camera movement in generated motions' is only mentioned in the conclusion; it should be stated earlier in the method section, since camera movement in the source videos is a key confound for the pseudo-label extraction.","section":"Sec. 6, Conclusions"}],"recommendation":"major_revision","confidential_remarks":"The dataset and code are not publicly released (no code link is mentioned beyond the project page), which may limit reproducibility; however, the manuscript does not claim a public release. The evaluation gap around the central scene-compatibility claim is the main concern; the authors should be encouraged to address it with quantitative projection-based metrics and a larger human/VLM study. The paper is otherwise within scope for a computer vision venue and the proposed task has practical value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this paper starts a genuinely new task — generating SMPL motion from a single scene image plus text — and backs it with a 300k-video dataset (HiC-Motion). That part is real and useful. The architecture is an assembled transformer diffusion model with CLIP text tokens and DINO image tokens as in-context conditions; the ablations show that configuration helps. If you work in human motion or video generation, this is a relevant result.\n\nBut the evaluation has a soft spot that matters. The central claim is that the motion 'projects naturally onto the scene image.' Yet the main quantitative metrics (FID, accuracy, diversity, multimodality) are computed after explicitly ignoring global orientation and translation — exactly the parameters that determine where the person stands in the scene. So those numbers measure pose quality and action recognition, not scene compatibility. The only direct evidence for scene alignment is the VLM study in Table 3, which averages 20 test videos, scores only the middle frame, and reports no inter-rater or repeat-run statistics. That is thin for the paper's headline claim. The stress-test note is on target.\n\nThere are also secondary weaknesses: the pseudo-GT labels from 4D-Humans could be noisy, and the held-out test set comes from the same pipeline, so the numbers are internally consistent but not independent validation. No code or dataset is released yet, which limits independent checks. Point estimates without variance across seeds are another minor concern.\n\nNone of this is fatal. The method is plausible and the qualitative results show real scene awareness, e.g. someone standing on a cliff or surfing. The ablations are sensible. The paper deserves a serious referee; it would come back with a request for a direct scene-alignment metric (e.g. measuring 2D projection error against ground-truth global parameters) and a larger VLM evaluation. I'd be happy to see it revised.","headline":"New task and large dataset are solid, but the main evaluation discards the scene-placement coordinates, so the central claim is only weakly supported.","tokens_in":13389,"tokens_out":2110,"would_cite":true,"duration_ms":19030,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Given a scene image and a text prompt, Move-in-2D generates a motion sequence that matches the text and projects naturally onto the scene.","keywords":["human motion generation","2D-conditioned motion generation","text-to-motion","scene-conditioned motion","diffusion transformer","SMPL body model","motion-guided video generation","HiC-Motion dataset"],"falsifier":"Collect 100 held-out videos with verified static cameras, manually mark the 2D positions of the person's feet and pelvis every 16 frames, generate a motion from the scene image and text, and compare the generated joints projected onto the image against those manual marks; a median projection error above roughly 20 pixels on trained actions would contradict the claim that the motion projects naturally onto the scene.","tokens_in":1772,"feed_emoji":"🏃","tokens_out":2054,"duration_ms":96469,"temperature":0.7,"pith_summary":"Move-in-2D introduces a new task: generating a human motion sequence from a scene image and a text prompt, so that the motion matches the text and lands naturally on the image when projected. The paper argues that a single 2D photo carries enough spatial and semantic layout information to condition motion, making 3D scene reconstruction unnecessary for many applications. To support this, the authors assemble HiC-Motion, a 300k-video corpus of single-person activities with auto-extracted SMPL body poses and background images. A diffusion transformer trained on this corpus with in-context conditioning on text and image tokens reports the best FID (44.639) and prompt accuracy (0.661) among the compared text- and scene-conditioned models. If this holds, it gives video generation systems a practical way to obtain scene-compatible motion without motion capture or 3D scanning.","feed_headline":"Move-in-2D generates human motion from a single photo and a prompt","feed_subtitle":"A 300k-video dataset and a diffusion transformer turn text plus one photo into scene-aligned human motion.","key_machinery":"The load-bearing machinery is a multi-conditional diffusion transformer. The motion representation is a sequence of SMPL body poses (SMPL is a standard skinned human body model): pose parameters $\\theta_b \\in \\mathbb{R}^{23\\times 6}$ for 23 body joints, a global orientation $\\theta_g \\in \\mathbb{R}^6$, and a camera translation $\\pi \\in \\mathbb{R}^3$, so the generated pose can be projected onto the scene image with a fixed-focal-length perspective camera. The text prompt is encoded into a single token and the scene image into patch tokens by a vision encoder that preserves spatial layout; all condition tokens are then concatenated with the noisy motion tokens (in-context conditioning) inside transformer blocks that use adaptive layer normalization for the diffusion timestep. This shared token space lets the model align motion with both modalities, while the two-stage training first learns diverse scene-aware motion from 300k videos and then fine-tunes on a subset emphasizing large motion and static backgrounds to separate body motion from camera-induced movement.","core_discovery":"The central claim is that 2D-conditioned human motion generation is feasible and useful: a diffusion model accepting a language-encoded text token and patch-level scene-image tokens, concatenated in-context with the noisy motion tokens, can output a 256-frame SMPL body-pose sequence plus a camera translation $\\pi \\in \\mathbb{R}^3$ that lets the motion be projected onto the background image with correct placement. Trained in two stages on HiC-Motion (the full corpus, then a 150k subset biased toward large motion and static backgrounds), the model achieves FID 44.639, accuracy 0.661, and diversity 26.027, beating text-only models, point-cloud-conditioned models, and a same-corpus text-only variant. The authors further show that feeding the generated motion to motion-guided video animators produces smoother, geometrically cleaner human video than unguided video diffusion, and that the in-context conditioning design outperforms cross-attention in ablations.","pith_inferences":["Because the held-out test set is labeled by the same automatic reconstruction pipeline used for training, part of the reported advantage could come from the model learning the reconstruction model's biases rather than from true scene understanding; an independent motion-capture test set would separate these.","The paper does not model camera movement and admits the two-pass video pipeline is not jointly optimized; a natural extension is to predict a full camera trajectory and to fine-tune the motion generator and video animator together, which may further improve temporal consistency.","The automated VLM evaluation uses only the middle frame of 20 generated videos, so it is a weak test of temporal scene alignment; a per-frame projection metric or a user study with more samples would give a stricter check.","A controlled comparison in which 3D-scene baselines are retrained on the HiC-Motion corpus, rather than only evaluated with back-projected depth, would isolate how much of the improvement is due to 2D conditioning versus dataset scale."],"forward_implications":["A single 2D photo plus a text prompt is enough to synthesize a motion sequence that can be re-projected onto the photo, so motion generation no longer requires 3D reconstruction or an existing driving motion.","The generated motions can serve as control signals for motion-guided video animators, producing videos with sharper human geometry and more consistent motion than unguided video diffusion.","Training on a large corpus of real-world videos with auto-extracted body poses substantially improves text-to-motion quality, since a same-corpus text-only model far outperforms its counterpart trained on smaller motion-capture datasets.","In-context conditioning, which concatenates text and scene tokens with motion tokens in a shared space, outperforms cross-attention for aligning generated motion with both modalities, and AdaLN timestep conditioning improves temporal smoothness.","The method covers large-dynamics activities and human-object interactions such as tennis, trampoline, climbing, horse riding, and petting a dog, because 2D scene images are far more abundant than annotated 3D scenes."],"supporting_citations":[{"why":"It supplies the pseudo ground-truth SMPL motion labels used both to train the model and to construct the held-out test set.","marker":"[14]"},{"why":"It encodes the scene image into spatial patch tokens, providing the layout information used for in-context conditioning.","marker":"[37]"},{"why":"It encodes the text prompt into a token used as the language condition in the diffusion transformer.","marker":"[43]"},{"why":"It supplies the diffusion-transformer backbone and the in-context conditioning design from which the model's architecture is adapted.","marker":"[39]"},{"why":"It serves as the primary text-only baseline and as the architecture underlying MDM+, which the paper retrains on its dataset for comparison.","marker":"[45]"},{"why":"It is a latent-space text-to-motion diffusion baseline used in the quantitative comparison.","marker":"[8]"},{"why":"It is the closest previous approach, conditioning motion on text and 3D point clouds, and is used as an evaluation and architectural reference.","marker":"[49]"},{"why":"It is the downstream motion-guided video animator used to show that generated motions improve human video quality.","marker":"[60]"}],"fun_headline_variants":["Move-in-2D: human motion from a photo and text","Scene-image conditioned human motion generation","Diffusion model turns photo and prompt into motion","2D-conditioned motion: from stills to human poses","Generate scene-aligned human motion from one image"],"cache_read_input_tokens":15488,"weakest_assumption_plain":"The load-bearing premise is that the automatically extracted body poses from hundreds of thousands of internet videos are accurate enough to serve as both training targets and evaluation ground truth; if these pseudo-labels drift or conflate camera motion with body motion, the model learns a distorted motion distribution and the reported scores inherit that distortion.","fun_headline_variants_meta":{"raw":{"variants":["Move-in-2D: human motion from a photo and text","Scene-image conditioned human motion generation","Diffusion model turns photo and prompt into motion","2D-conditioned motion: from stills to human poses","Generate scene-aligned human motion from one image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000362,"raw_usage":{"total_tokens":1931,"prompt_tokens":899,"completion_tokens":1032,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":957}},"tokens_in":515,"tokens_out":1032,"duration_ms":7965,"temperature":1.0,"reasoning_tokens":957,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:20:07.708427+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect 100 held-out videos with verified static cameras, manually mark the 2D positions of the person's feet and pelvis every 16 frames, generate a motion from the scene image and text, and compare the generated joints projected onto the image against those manual marks; a median projection error above roughly 20 pixels on trained actions would contradict the claim that the motion projects naturally onto the scene.","supporting_citations":[{"cited_title":"Humans in 4D: Reconstructing and tracking humans with transformers","cited_arxiv_id":null,"evidence_quote":"It supplies the pseudo ground-truth SMPL motion labels used both to train the model and to construct the held-out test set."},{"cited_title":"Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever","cited_arxiv_id":null,"evidence_quote":"It encodes the text prompt into a token used as the language condition in the diffusion transformer."},{"cited_title":"Scalable diffusion models with transformers","cited_arxiv_id":null,"evidence_quote":"It supplies the diffusion-transformer backbone and the in-context conditioning design from which the model's architecture is adapted."},{"cited_title":"Human motion diffu- sion model","cited_arxiv_id":null,"evidence_quote":"It serves as the primary text-only baseline and as the architecture underlying MDM+, which the paper retrains on its dataset for comparison."},{"cited_title":"Executing your commands via motion diffusion in latent space","cited_arxiv_id":null,"evidence_quote":"It is a latent-space text-to-motion diffusion baseline used in the quantitative comparison."},{"cited_title":"Humanise: Language-conditioned hu- man motion generation in 3d scenes","cited_arxiv_id":null,"evidence_quote":"It is the closest previous approach, conditioning motion on text and 3D point clouds, and is used as an evaluation and architectural reference."},{"cited_title":"Champ: Controllable and consistent human image animation with 3d parametric guidance","cited_arxiv_id":null,"evidence_quote":"It is the downstream motion-guided video animator used to show that generated motions improve human video quality."}],"review_version":1}