{"id":"0e592e1f-3472-4821-82ad-d0b18ac52da9","arxiv_id":"2412.10219","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A diffusion inpainting model finetuned on video frame pairs with noisy captions and pose conditioning gives text-controllable identity-preserving edits of people in new scenes.","lead":"This paper trains a Stable Diffusion model to take a single photo of a person and insert them into a new scene while following a text instruction and a target pose. The authors add weak captions from a vision language model to pose data, and report improved person object interactions in generated images.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is not supported by Table 3: adding text to pose yields no measured gain in person-object interaction plausibility (33% vs 33%), so the reported improvement is attributable to pose alone.","rationale":"The reader's weakest assumption concerns the training data validity (same person, plausible pose change, valid captions). That is a real risk, but the more direct load-bearing weakness is internal: the paper's own Table 3 shows no interaction-quality benefit from adding text to pose, which is the exact comparison needed for the headline 'combining text with pose improves person-object interactions.' This is an internal consistency issue rather than a question of external consensus. The dataset construction and baseline reimplementation are valuable, and the issue is addressable: either a proper statistical reanalysis shows a significant interaction benefit, or the paper's claims must be narrowed to identity preservation and to pose-driven interaction gains. I would keep the conditional verdict but make the condition explicit: the central interaction claim needs statistical support from per-scene paired ratings, not just averaged percentages. My concern partially overlaps with the reader's, since noisy captions could explain why text adds no interaction signal, but the failure is evident from the reported numbers even if the captions were perfect.","tokens_in":11017,"tokens_out":6227,"duration_ms":59252,"concrete_test":"Obtain the per-scene binary ratings behind Table 3 (or rerun the user study with at least 50 scenes per configuration) and compute a paired comparison between Img-Pose-Text and Img-Pose on interaction plausibility, and between Img-Pose and Img-Only. Report the differences with confidence intervals and a McNemar or sign test. If the text+pose versus pose-only difference is not significantly positive, revise the central claim to attribute object-interaction gains to pose conditioning, or present text as improving identity preservation rather than interaction quality.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline contribution—'Combining the weak supervision from noisy captions, with robust 2D pose improves the quality of person-object interactions'—is not established by the paper's own quantitative evidence. Table 3 reports interaction plausibility of 24% for Img-Only, 24.5% for Img-Text, 33% for Img-Pose, and 33% for Img-Pose-Text. The only gain over the image-only baseline is pose conditioning; adding the weak text caption to pose changes the interaction score by exactly zero. The combination does improve identity preservation on interaction scenes (41% vs 25%), but that is a different claim from the 'quality of person-object interactions' stated in the abstract and conclusion. The conclusion's phrase 'significant improvements ... particularly in scenes involving human-object interactions' is not supported by any significance test, and the 33% vs 24% gap has no confidence interval. Because the central scientific claim is an interaction effect between text and pose, a comparison that shows no marginal interaction benefit is a direct, unaddressed contradiction rather than a missing extra. The appendix's documented caption failures (left/right reversal, reverse order) make it plausible that weak text does not add interaction signal, but the concern here is internal to the reported tables and does not depend on dataset-quality assumptions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses the task of inserting a person from a reference image into a novel scene with controllable non-rigid edits. The authors fine-tune a Stable Diffusion inpainting model on pairs of video frames, using GPT-4V-generated scene-difference captions and RTMPose 2D skeletons as additional conditioning signals. They compare four configurations (image-only, image-text, image-pose, image-pose-text) using FID, PCKh, and a human user study, and they reimplement the Kulal et al. baseline on public data. The paper also contributes a processed dataset and caption annotations.","tokens_in":11241,"tokens_out":4574,"duration_ms":41283,"significance":"If the central claim were established, this would be a useful step: combining weak text supervision with pose conditioning for identity-preserving, non-rigid human insertion, plus a public-data reimplementation of a previously closed-source method. The paper has clear strengths: it reimplements Kulal et al. on public data, documents captioning failures honestly, and provides qualitative evidence that text and pose together enable controllability. However, the headline claim that weak captions plus pose improves person-object interaction quality is not supported by the paper's own quantitative table, and the user study is not reported with enough statistical detail to distinguish the observed differences from noise. The contribution is therefore promising but not yet demonstrated at the level claimed.","major_comments":[{"comment":"The abstract and conclusion claim that combining weak captions with pose improves the quality of person-object interactions, but Table 3 reports interaction plausibility of 33% for both Img-Pose and Img-Pose-Text, with Img-Text at 24.5% and Img-Only at 24%. The only measurable gain over the image-only baseline comes from pose conditioning; text adds zero marginal improvement to interaction plausibility. The identity-preservation gain of Img-Pose-Text (41%) over Img-Pose (25%) is a different result and should not be presented as an interaction-quality improvement. The central claim needs to be either supported with a statistically significant contrast between Img-Pose-Text and Img-Pose on interaction plausibility, or revised to state that pose drives interaction quality while text improves identity preservation.","section":"§4.4, Table 3"},{"comment":"The user study is not reported with enough statistical detail to support the quantitative claims. Only eight raters and 50 scenes per configuration are used, answers are binary, and no confidence intervals, significance tests, or inter-rater agreement measures are given. With these numbers, a difference of 33% versus 24% (roughly 4-5 binary responses out of 50) can easily be within sampling noise. The authors should report per-rater variation, compute confidence intervals or a paired significance test, and ideally increase the number of scenes or raters for the key interaction comparison.","section":"§4.1, §4.4"},{"comment":"The paper claims superior performance compared to baselines such as ControlNet, MASACtrl, and PIDM, but the actual comparisons to these baselines are qualitative only. There is no quantitative evaluation of these baselines on the same scenes, and no user-study ratings for them. To substantiate the third contribution, the authors should evaluate at least one state-of-the-art baseline under the same identity-preservation and controllability protocol used in Tables 2 and 3.","section":"§4.3, Figure 3"},{"comment":"The captioning pipeline is a central contribution, but its error rate is never quantified. The appendix documents left/right reversals, reversed ordering, and occasional misidentification of objects, yet no human evaluation of caption accuracy or an analysis of how caption errors affect controllability is provided. A small human-annotated subset with caption correctness statistics would strengthen the claim that weak captions are usable and would let readers assess the effect of caption noise.","section":"§3.3, A.3"},{"comment":"The frame-pair filtering criteria are described qualitatively: minimum pose distance is said to be equivalent to the length of the pose skeleton's shoulder-to-head distance, and histogram similarity is said to have minimum and maximum thresholds, but no numerical values are reported. Since the data-quality argument is load-bearing for the method, the exact thresholds and a sensitivity analysis should be reported for reproducibility.","section":"§3.2, §3.4"}],"minor_comments":[{"comment":"The sentence 'These methods maintain subject identities, however are non-rigid edits' appears to be missing a negation; the intended meaning is likely that these methods are not non-rigid edits.","section":"§2.3"},{"comment":"The text says 'Responses are averaged for 4 users each into the tables below,' but earlier it says eight raters were recruited; clarify whether each scene was rated by four or eight raters.","section":"§4.4"},{"comment":"The reference to 'PIDM [ ?]' is an unresolved placeholder and should be replaced with the actual reference [3].","section":"Appendix A.5"},{"comment":"The caption states that 'Adding text or pose to image conditioning improves identity preservation,' but the Img-Text row shows identity at 55%, which is lower than the Img-Only value of 61%; adjust the caption to match the numbers.","section":"Table 2"},{"comment":"The caption uses 'our's' where 'ours' is intended.","section":"Figure 3 caption"},{"comment":"The statement that 'image-text and image-pose models achieve the best results' conflicts with the user-study tables, where image-pose-text is highest on identity in both Table 2 and Table 3; rephrase for clarity.","section":"§4.6"},{"comment":"The paper states that the dataset is open-source, but no URL or release instructions are provided; an availability statement should be added.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations and includes useful failure cases, and the public-data reimplementation is a valuable artifact. However, the main quantitative claim needs rework: Table 3 does not show any marginal benefit of text over pose for interaction plausibility, and the user study lacks the statistical reporting needed to support the comparisons. I would be willing to review a revised version that either provides a statistically sound demonstration of the text-plus-pose interaction effect, or reframes the central claim to match the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, the engineering is real: the authors re-implemented Kulal et al.'s affordance-aware human insertion on public data, built a dataset of roughly 13,500 captioned video frame pairs, and documented the pipeline clearly. That is a useful contribution on its own, because the original model is proprietary and no checkpoints exist. Second, the paper's central claim as written is not supported by its own numbers. The abstract says combining noisy captions with robust 2D pose improves the quality of person-object interactions. Table 3 shows the interaction plausibility is 33% for image+pose and 33% for image+pose+text. Adding text to pose buys exactly zero on the interaction axis. The only measured gain from the full combination is identity preservation (41% vs 25%), which is a different claim from what the abstract states. The conclusion's phrase \"significant improvements\" is not backed by any significance test; the study has 8 raters, 50 scenes per cell, binary answers, no confidence intervals, and no inter-rater agreement.\n\nWhat the paper does well: the public-data retraining of Kulal et al. is valuable, and the qualitative results in Figures 1, 2, and 5 are often plausible. The GPT-4V captioning pipeline is documented honestly, including failure modes (left/right reversal, reversed order), and the limitations section is frank about identity rates in the 55-68.5% range. The choice of RTMPose over OpenPose is sensible, and the filtering criteria are described in enough detail to reproduce.\n\nSoft spots beyond the headline: the citation list has material errors—[16] and [3] share the same arXiv ID (2304.14406), the DreamBooth reference has the wrong authors, and Charades is cited with a paper that is not the dataset reference. No code, model, or dataset artifacts are released, which limits reproducibility. The FID/PCKh numbers in Table 1 are acknowledged as imperfect, but the paper still leans on them in the ablation rather than a task-level metric.\n\nWho this is for: anyone working on identity-preserving human insertion or building weakly supervised video-to-text training sets. The dataset and reimplementation deserve referee time. But the current version overclaims the interaction effect; the abstract and conclusion should be rewritten to claim identity improvement, or the authors need to show a setting where text actually moves the interaction number. A serious referee could help fix this, so I would send it to review, but I would expect major revision.","headline":"Solid engineering and a valuable public-data baseline, but the paper's headline claim—that text improves person-object interactions—is contradicted by its own Table 3.","tokens_in":11826,"tokens_out":2740,"would_cite":true,"duration_ms":24672,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fine-tuning a Stable Diffusion inpainting model on filtered video-frame pairs with GPT-4V-generated scene-difference captions and 2D pose conditioning yields identity-preserving, text-controllable non-rigid edits in…","keywords":["diffusion models","image editing","identity preservation","non-rigid pose edits","pose conditioning","text conditioning","multimodal captions","person-object interaction"],"falsifier":"If, in a held-out test set of pairs whose captions describe a leftward or rightward movement, the model consistently mirrors the opposite direction (following its captioning errors), the claim that it learned the text-to-edit mapping from the noisy captions would be falsified; equivalently, a larger preregistered user study that finds no significant difference between image-pose and image-pose-text on object-interaction plausibility would falsify the central thesis that pose+text improves person-object interactions.","tokens_in":10773,"feed_emoji":"🖼️","tokens_out":7774,"duration_ms":69829,"temperature":0.7,"pith_summary":"The paper tries to establish that a diffusion model can insert a single photo of a person into a new scene and change that person's pose in ways controlled by a short text prompt, while keeping the person's identity intact. The training signal is weak and cheap: pairs of frames from human-action videos, with the pose change between frames described automatically by a multimodal LLM rather than by human annotators. The authors claim that adding 2D pose skeletons alongside these noisy text captions materially improves the model's handling of person-object interactions, which they identify as the hardest case of identity preservation. If the claim holds, user-controllable, identity-preserving image editing can be learned without expensive paired annotation, extending to scenes outside the training distribution. The paper also contributes an open-source dataset of 13,487 captioned video-frame pairs and reimplements the leading prior baseline on public data.","feed_headline":"Pose plus noisy captions control non-rigid person edits","feed_subtitle":"Fine-tuned Stable Diffusion inserts one person into a new scene and re-poses them from a text prompt.","key_machinery":"The key mechanism is joint multimodal conditioning in an inpainting diffusion framework. A masked target frame provides the scene to insert into; a segmented reference crop carries identity through CLIP-image cross-attention; a GPT-4V-generated scene-difference caption enters through the CLIP text encoder; and a 17-joint 2D skeleton, flattened to 51 values and projected through a linear layer, carries pose. These three embedding streams are concatenated into one (batch, 335, 768) tensor that conditions the U-Net, and classifier-free guidance is implemented with separate unconditional representations for image, text, and pose. The other load-bearing component is the data pipeline: keyframes are sampled per video using a minimum pose distance (shoulder-to-head length) and histogram-similarity bounds, and RTMPose is used to filter scenes with a single full skeleton and to produce the conditioning poses.","core_discovery":"The central discovery is that multimodal conditioning—a reference image of the person, a text caption describing the pose change, and a 2D pose skeleton—can be fused in a Stable Diffusion inpainting model fine-tuned on self-supervised video-frame pairs, and that this fusion yields identity-preserving, text-controllable non-rigid edits on in-the-wild images. On scenes without object interactions, the combined image-pose-text model achieves the highest identity-preservation rating (68.5%) and strong control-signal adherence (51%) in a user study, at the cost of slightly lower FID than image-only conditioning. On scenes with person-object interactions, where identity preservation rates drop to 25-50%, pose and text jointly produce the most plausible interactions (33%), confirming the paper's thesis that weak caption supervision plus robust pose conditioning improves person-object interaction quality. The paper also documents the limits: identity preservation is not always achieved, long narrow objects are brittle, and automatically generated captions contain left/right and ordering errors.","pith_inferences":["If the left/right and reverse-order caption errors are the main noise source, then improving caption fidelity (for example, by providing frame order explicitly to the LLM, or by filtering captions against the measured pose delta) could push identity preservation and control adherence above the reported 68.5% and 51%.","The same joint conditioning might transfer to articulated non-human subjects (animals, robots) as long as a pose skeleton and paired frames are available, making the method a general recipe for controllable subject insertion.","The authors hypothesize that more accurate captions would improve the image-text model; a testable extension is to use the pose delta to reject captions that disagree with the measured motion, which could raise interaction plausibility beyond the current 33% ceiling.","The method's dependence on a single pretrained captioner suggests a self-training loop: generate an edit, caption the difference with the same LLM, and add the pair back into the training set—whether such a loop converges or amplifies errors is untested."],"forward_implications":["A user can take an unseen photo of a person and a target scene, and request multiple different non-rigid edits (turn, lunge, raise arms) by changing only the text prompt, without retraining or per-edit tuning.","Weakly supervised video-frame pairs suffice to train this controllability: 5,787 captioned Kinetics videos plus annotated pairs from NTU-RGBD, Charades, and Fit3D produce the identity-preserving edits, meaning the approach can scale to unlabeled video.","The best identity-preservation and control adherence come from combining reference image, text, and pose; pose alone or text alone is worse, establishing the joint-conditioning recipe as the method's core.","FID is an imperfect proxy for this task: the image-only model wins on FID yet performs worse on identity and control in user studies, arguing for control-aware evaluation of editing models.","Open-sourcing the captioned dataset and the public reimplementation lets other methods be compared on identity-preserving non-rigid editing without the closed data used by prior work."],"supporting_citations":[{"why":"Supplies the inpainting diffusion formulation, masking and augmentation procedures, and the image-only baseline that this work re-implements on public data.","marker":"[16]"},{"why":"The multimodal LLM (GPT-4V) used in a 10-shot manner to generate scene-difference captions that provide the weak text supervision.","marker":"[6]"},{"why":"RTMPose provides the 2D 17-joint skeletons used both to filter video frames and to condition pose during training and inference.","marker":"[14]"},{"why":"Kinetics-700 is the main source of action-rich human-centric videos; 5,787 captioned videos are retained after filtering.","marker":"[8]"},{"why":"NTU-RGB+D supplies 7,700 annotated image pairs and is processed with the looser original pipeline of the baseline.","marker":"[19]"},{"why":"MASACtrl serves as a tuning-free text-driven non-rigid edit baseline; the comparison shows brittleness on in-the-wild data.","marker":"[7]"},{"why":"PIDM serves as the pose-conditioned baseline whose performance is brittle outside its fashion-domain training data.","marker":"[3]"},{"why":"ControlNet is used as a text-controllable baseline that the paper notes does not preserve identity.","marker":"[24]"}],"fun_headline_variants":["Pose and text conditioning outperform image-only for identity preservation","Fine-tuned diffusion inserts people into scenes with pose and prompt control","Noisy captions plus 2D pose improve person-object interaction realism","Multimodal cues from video pairs enable non-rigid human edits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training pairs from the same video actually show the same person performing a plausible pose change, so that the masked target, reference crop, and auto-generated caption describe one coherent edit; if filters let through identity switches or captions that reverse the motion direction, the learned text-to-edit mapping is corrupted.","fun_headline_variants_meta":{"raw":{"variants":["Pose and text conditioning outperform image-only for identity preservation","Fine-tuned diffusion inserts people into scenes with pose and prompt control","Noisy captions plus 2D pose improve person-object interaction realism","Multimodal cues from video pairs enable non-rigid human edits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000352,"raw_usage":{"total_tokens":1915,"prompt_tokens":937,"completion_tokens":978,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":903}},"tokens_in":553,"tokens_out":978,"duration_ms":10407,"temperature":1.0,"reasoning_tokens":903,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:12:27.665083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If, in a held-out test set of pairs whose captions describe a leftward or rightward movement, the model consistently mirrors the opposite direction (following its captioning errors), the claim that it learned the text-to-edit mapping from the noisy captions would be falsified; equivalently, a larger preregistered user study that finds no significant difference between image-pose and image-pose-text on object-interaction plausibility would falsify the central thesis that pose+text improves person-object interactions.","supporting_citations":[{"cited_title":"The Bounds of Mediated Communication","cited_arxiv_id":"2303.06244","evidence_quote":"RTMPose provides the 2D 17-joint skeletons used both to filter video frames and to condition pose during training and inference."},{"cited_title":"Patchy particles by self-assembly of star copolymers on a spherical substrate: Thomson solutions in a geometric problem with a color constraint","cited_arxiv_id":"1907.08103","evidence_quote":"Kinetics-700 is the main source of action-rich human-centric videos; 5,787 captioned videos are retained after filtering."},{"cited_title":"Ntu rgb+d: A large scale dataset for 3d human activity analysis","cited_arxiv_id":null,"evidence_quote":"NTU-RGB+D supplies 7,700 annotated image pairs and is processed with the looser original pipeline of the baseline."},{"cited_title":"Physics-informed radial basis network (PIRBN): A local approximating neural network for solving nonlinear PDEs","cited_arxiv_id":"2304.06234","evidence_quote":"MASACtrl serves as a tuning-free text-driven non-rigid edit baseline; the comparison shows brittleness on in-the-wild data."}],"review_version":1}