{"id":"83af5f00-c3c9-495a-a3a5-43bc98763dd6","arxiv_id":"2412.10275","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TIV-Diffusion adds object-centric slot alignment to a diffusion-based image-to-video generator and reports improved alignment and temporal-consistency metrics on MNIST, CATER, and Bridge datasets.","lead":"Researchers propose TIV-Diffusion, a diffusion-based system that turns a single starting image plus a text description of motion into a short video by first splitting the image into object slots and aligning each slot with words from the caption.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim of precise text-to-object motion control is never directly measured; Tables 1-2 use global FID/FVD, and the slot-to-text alignment in Eq. 6 is unsupervised, so improved alignment over baselines remains an assumption.","rationale":"The reader's verdict is CONDITIONAL, with the weakest assumption being that slots genuinely correspond to objects and stay aligned with text. I agree that the fixed slot count on CATER-GEN-v2 is a real risk, but the more load-bearing issue is that the paper's headline distinction, precise text-to-object motion alignment, has no quantitative test at all. FID/FVD are quality and distribution metrics and can improve even if the wrong object moves, because a sharp, temporally consistent video can score well on these datasets. Thus the improvement seen in Table 2 when adding ODFM, while consistent, does not establish that alignment is the mechanism. This is reinforced by the architecture: Eq. 6 is a residual cross-attention from slots to the full caption; no one-to-one slot-word objective or object-label supervision is used, so 'aligning textual features with each object individually' is not enforced. Table 4 also shows only small differences across slot counts, so tuning K on the evaluation set does not provide strong evidence that slots map to the actual objects. I would not reject the paper: the method is clearly described, the quality gains are plausible, and the proposed object-level controllability test is feasible on CATER. I would keep the CONDITIONAL verdict but make release of code and a direct alignment metric explicit conditions. This is why I mark agreement as partial: the reader's slot-correspondence concern is real, but even if slots were perfect, the central claim would still be unmeasured.","tokens_in":17720,"tokens_out":6231,"duration_ms":52371,"concrete_test":"Construct an object-level controllability metric on CATER-GEN-v2 test text-image pairs: using ground-truth object attributes and action labels, automatically determine (1) whether the object named in the caption is the one that actually moves in the generated video, and (2) whether the described action (rotate/contain/pick-place/slide) is executed. Report this accuracy for Ours, Ours w/o, MAGE, MAGE+, and Seer over the same test split with multiple seeds. If Ours is not clearly and statistically better than Ours w/o on this metric, the claimed benefit of the object-centric alignment module is not supported; if Ours is better than baselines on FVD but not on this metric, the 'precise control' claim should be separated from the quality claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TIV-Diffusion's distinctive claim is that object-centric textual-visual alignment yields precise control and better text-to-object motion alignment than MAGE/MAGE+/Seer. The quantitative evidence in Tables 1 and 2 is FID, FVD, SSIM, PSNR, and LPIPS, which are aggregate distribution or similarity scores that do not measure whether the text-specified object moved, or whether the motion matched the caption. The qualitative Figures 4-7 show selected successes, not error rates. In addition, Sec. 3.4 does not actually enforce object-centric alignment: slots are pre-trained by reconstruction (Eq. 4), Eq. 6 lets every slot attend to the whole caption, and the Gumbel-Softmax selection (Eqs. 7-9) is trained only with reconstruction loss (Eq. 10). Nothing ties a slot to an object identity or to a specific noun phrase, and the slot budget K+1=6 cannot cover the 3-8 objects in CATER-GEN-v2 (plus background). Thus even if the FVD gains are real, the mechanism responsible, and the central claim of improved text-to-object alignment, is untested. The paper's own Table 4 shows K=5 best by a very small margin (FID 6.67 vs 6.81 at K=11), so the slot-count evidence is not strong enough to carry the object-correspondence assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TIV-Diffusion, an autoregressive diffusion framework for text-driven image-to-video generation (TI2V). The model fuses text and image embeddings via cross-attention and injects them through SPADE modulation, and adds an object-centric module that extracts slots from the first frame with a Slot Attention encoder, augments each slot with text via cross-attention, and adaptively incorporates slot features into the U-Net through Gumbel-Softmax selection. Experiments are conducted on modified MNIST variants, CATER-GEN-v1/v2, and the Bridge dataset, reporting state-of-the-art FID/FVD and improved qualitative results for object preservation and motion alignment.","tokens_in":18000,"tokens_out":5974,"duration_ms":52833,"significance":"If the claims hold, the object-centric conditioning mechanism could be a useful contribution to the TI2V literature, targeting a real failure mode (object disappearance and deformation during text-specified motion). The paper's ablations are a strength: Tables 2, 5, 6, and 7 consistently show that adding the slot module and its associated components improves quantitative scores across MNIST, CATER, and Bridge, which supports the mechanism's practical value. However, the paper's central claim of 'precise control' and improved 'text-to-object motion alignment' is not directly quantified; the reported metrics are global video-quality scores, and the qualitative controllability evaluation in Supplementary A.3 is illustrative rather than measured. The paper also does not provide a public code release or detailed reproducibility statement, which limits verification of the training and evaluation pipeline.","major_comments":[{"comment":"The central claim that TIV-Diffusion provides 'precise control' and improved text-to-object motion alignment is not supported by the quantitative metrics. FID, FVD, SSIM, PSNR, and LPIPS are global distributional or similarity measures; they do not test whether the object named in the caption is the one that moves, whether the motion direction and magnitude match the caption, or whether the object's identity is preserved under occlusion. The qualitative figures (Figs. 4-7) and the controllability examples in Supplementary A.3 (Figs. 14-15) are selected successes and do not provide an error rate. For CATER datasets, ground-truth object trajectories and action labels are available; please add a quantitative controllability metric, e.g., top-1 accuracy of 'which object moved' and a trajectory-matching score against the caption, or report the success rate of generating the commanded action over the full test set. Without such a metric, the paper's main thesis remains untested.","section":"Sec. 4.3, Tables 1 and 2"},{"comment":"The proposed 'object-centric textual-visual alignment' does not contain any mechanism that ties a slot to a specific noun phrase or to a semantically meaningful object. The Slot Attention encoder is pre-trained only by reconstructing the initial frame, Eq. (4), with no object supervision. Then Eq. (6) is a standard cross-attention in which each slot attends to the entire text caption, and the final training loss, Eq. (10), is a pure frame-reconstruction loss. Therefore, the correspondence between slots and objects is assumed, not learned or enforced. This assumption is especially problematic for CATER-GEN-v2, which contains 3 to 8 objects, while the slot budget is K+1=6 (and K=5 is chosen as optimal in Table 4); with K=3, the same FID of 6.67 is obtained, further weakening the claim that the slot count reflects object count. Please provide explicit evidence of slot-to-object correspondence (e.g., slot-to-mask IoU on CATER), a slot-to-text alignment metric, or add a training objective that supervises the claimed alignment.","section":"Sec. 3.4, Eqs. (4)-(10), Table 4"},{"comment":"The 'state-of-the-art on existing datasets' claim is compromised by the modification of the three MNIST datasets. The paper states that 'we slightly modify these three datasets' to enforce uniform motion, and MAGE is reproduced on the modified versions. However, the original MNIST benchmarks are the standard evaluation, and the modification changes the task distribution (e.g., removing speed irregularities at boundaries). The comparison with TVP on the Single dataset uses results from the original paper, so it is not apples-to-apples with the modified setting. Please either report results on the original unmodified MNIST datasets, or explicitly state in the abstract and conclusion that the reported state-of-the-art applies only to the modified versions. At minimum, the quantitative comparison on MNIST should include results on the original datasets for both MAGE and TIV-Diffusion.","section":"Sec. 4.1, Table 1"},{"comment":"The hyperparameters K (number of slots) and T (denoising timesteps) appear to be selected on the evaluation sets: Table 4 selects K on CATER-GEN-v2, and Table 3 selects T on CATER-GEN-v1. This is test-set tuning and can inflate the reported numbers. The FID differences in Table 4 are also very small (6.67 vs 6.67 vs 6.72 vs 6.81), so the 'optimal' K=5 is not robustly justified. Please either fix K and T before seeing the test set, or use a held-out validation split to select them, and report the validation-based choice. This would also address the concern that the slot budget does not match the variable object counts in CATER-GEN-v2.","section":"Sec. 4.3 and Supplementary A.5, Tables 3 and 4"}],"minor_comments":[{"comment":"There is a typo: 'TIV-Diffuion' should be 'TIV-Diffusion'.","section":"Abstract"},{"comment":"The heading 'Quantitive Results' should read 'Quantitative Results'.","section":"Sec. 4.3 heading"},{"comment":"The module name is misspelled as 'Object Distanglement Fusion Module'; it should be 'Object Disentanglement Fusion Module'.","section":"Fig. 2 and Sec. 3.4"},{"comment":"The captions and text for Tables 6 and 7 are inconsistent. Table 6's caption says 'Slot attention encoder' but the text describes replacing the Image Encoder with a convolutional layer and labels the row 'Ours Conv'; Table 7's caption says 'Ablation results of ConvGRU' but the text says the ConvGRU is replaced with the Slot Attention encoder. Please align the captions, text, and row labels.","section":"Supplementary A.5, Tables 6 and 7"},{"comment":"The notation for the Gumbel-Softmax selection is under-specified: the normalization axis is stated, but the ranges of indices i, j, and the role of the straight-through estimator in Eq. (8) during inference are not clarified. Please state explicitly whether Gumbel noise is used at inference time or only during training.","section":"Eq. (7)-(9)"},{"comment":"The paper does not provide a code repository, data splits for the real-world datasets (Weizmann, KTH, Bridge), or seeds. Adding these would substantially improve reproducibility, especially given the dataset modifications in Sec. 4.1.","section":"General reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem and contains a substantial set of experiments and ablations. However, the main scientific claim is about controllability and object-centric alignment, and the current evaluation does not measure those constructs. The slot-attention mechanism is plausible but the evidence for slot-object correspondence is indirect. These issues are fixable with additional metrics and comparisons, so I recommend major revision rather than rejection. I would also encourage the editor to verify whether the modified MNIST datasets are disclosed clearly enough in the abstract, as the SOTA claim could mislead readers if overlooked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's a quick take on TIV-Diffusion. It's a competent, well-put-together TI2V paper, and the object-centric slot module clearly helps on the benchmarks they actually use. But the paper's headline claim about 'precise control' over object movement is more asserted than measured.\n\nThe new thing is the specific integration: Slot Attention pretrained by reconstruction, per-slot cross-attention with the caption, Gumbel-Softmax slot selection, and SPADE scale-offset modulation inside an autoregressive diffusion generator. Each piece is familiar, but I don't know a prior TI2V paper that puts them together this way, and the ablations consistently show the slot module improves FID/FVD across CATER-v1/v2 and the MNIST variants. The qualitative comparisons, especially the occlusion cases in Figs. 4 and 6, show real differences from MAGE. So the engineering contribution is genuine.\n\nThe soft spots are mostly about evaluation. First, the 'precise control' claim is not directly quantified. FID/FVD/SSIM/LPIPS are aggregate similarity scores; they don't tell you whether the cone moved to the right coordinate or whether the right object moved. The supplementary has a controllability evaluation (Figs. 14-15) but it's qualitative. A scalar metric for text-to-object alignment would make the central claim testable. Second, the MNIST 'SOTA' numbers are on datasets the authors modified to enforce uniform motion. They fairly re-train/reproduce MAGE on the modified data, so the comparison is consistent, but the modified benchmarks are not the original ones, and the reader should weigh the CATER results more heavily. Third, no error bars and no code release. Finally, the slot-count ablation shows K=5 over K=11 is a small margin (FID 6.67 vs 6.81), so the 'each slot corresponds to an object' story is plausible but not strongly pinned down by that table.\n\nNone of this is load-bearing. The internal ablation (with vs without slots) is consistently positive, and that's the core empirical claim. The paper is a solid advance for the TI2V subfield, and it deserves a serious referee. My advice to the editor: send it to review, but ask the authors for a direct controllability metric, error bars, and code/checkpoints. If they can provide that, the paper is a nice contribution.","headline":"Solid TI2V engineering: the slot module reliably helps on standard benchmarks, but the 'precise control' claim needs a direct metric before the paper earns its headline.","tokens_in":18561,"tokens_out":3420,"would_cite":true,"duration_ms":615307,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TIV-Diffusion reports that decomposing the starting frame into object slots and aligning each slot with the caption lets a diffusion model move the right object with less deformation and disappearance, outscoring prior text-image-to-video…","keywords":["text-driven image-to-video generation","object-centric representation","Slot Attention","diffusion model","autoregressive video generation","cross-modal alignment","Gumbel-Softmax"],"falsifier":"Hold the slot count at five on CATER-GEN-v2 and test captions in scenes with eight objects, then compare whether the object named in the caption is the one that actually moves; if compliance collapses once the object count passes the slot budget, the claim that slots carry object-text alignment fails.","tokens_in":17525,"feed_emoji":"🎬","tokens_out":11330,"duration_ms":93610,"temperature":0.7,"pith_summary":"TIV-Diffusion sets out to show that text-driven image-to-video generation improves when the model first decomposes the given first frame into separate object slots and aligns the text caption with each slot individually, instead of treating the image as one global conditioning signal. The framework couples an autoregressive diffusion video generator with a Slot Attention encoder: slots extracted from the initial frame are enriched by cross-attending to the caption, and the text-enhanced slots are fed into the U-Net during denoising alongside a fused text-image embedding. On the paper's MNIST and CATER experiments, this object-centric alignment yields better perceptual-quality scores and better text-to-motion consistency than prior TI2V methods, and it specifically reduces the failure modes where the wrong object moves, the motion does not match the text, or an object deforms or vanishes when frames overlap. A sympathetic reader would care because the proposed mechanism is modular and resource-friendly: it achieves these gains with an autoregressive frame-by-frame generator rather than a large full-video model.","feed_headline":"Object slots keep text-driven videos from losing objects","feed_subtitle":"Splitting the first frame into per-object slots aligned with the caption sharpens motion control and cuts object loss.","key_machinery":"The load-bearing mechanism is the object-centric textual-visual alignment realized by Slot Attention, an attention process that iteratively partitions image features into a fixed set of slots. A Slot Attention encoder, pre-trained by reconstructing the first frame, produces K object slots plus a background slot, each meant to hold one object's attributes. Each slot is then refined by cross-attending to the caption, so words describing a target object are bound to the slot that carries that object's identity; the result is denoted M. The text-enhanced slots are linearly projected and injected into every upsampling residual block of the U-Net through a Gumbel-Softmax selection (Eqs. 7–9), letting the model pick the slot relevant to the current motion instruction while keeping a residual pathway for object identity. In parallel, the caption and image-patch embeddings are fused by cross-attention and applied as SPADE scale-offset modulation (Eq. 2), which is what carries the global appearance and motion trajectory. The two pathways together give the generator both a global sense of which object goes where and an object-level memory that resists deformation and disappearance.","core_discovery":"The paper's central claim is that object-centric textual-visual alignment—learning a fixed set of object slots from the reference image and aligning each slot to the caption—is what allows a diffusion-based TI2V model to move the correct objects along the described trajectories while keeping those objects intact. Concretely, TIV-Diffusion reports lower Fréchet Inception Distance (FID) and Fréchet Video Distance (FVD) than the strongest prior methods on CATER-GEN-v1 and CATER-GEN-v2 (for example, FID drops from 23.56 to 6.67 and FVD from 34.93 to 16.48 on CATER-GEN-v2), and similar gains on Single and Double Moving MNIST. The ablation without object-centric representations shows the slot pathway is doing the work: on CATER-GEN-v2 FID rises from 6.67 to 10.66 and FVD from 16.48 to 32.72 when the slots are removed. The paper also shows qualitative cases where prior methods move the wrong visually similar object or lose an object during overlap, while TIV-Diffusion keeps the object's attributes available through the slots.","pith_inferences":["The reported metrics mainly measure perceptual fidelity and distribution distance, not whether the text-referenced object is the one that moved; a direct trajectory-compliance metric computed against CATER's known object positions would be a stronger test of the paper's central controllability claim.","Because slots are extracted from the first frame only, the approach implicitly assumes the objects to be moved are present and identifiable at t=0; objects that enter later, or slots that should update as appearance changes, lie outside the current design.","The fixed slot budget is a capacity bottleneck for open-world use: CATER-GEN-v2 contains 3 to 8 objects while the tuned K is 5, so scenes with more objects require sharing or dropping slots; a dynamic slot-count mechanism would be the obvious follow-up.","The same per-slot text alignment could be lifted into other conditional U-Net pipelines, such as image editing or multi-object animation, wherever instance identity must survive cross-attention with language."],"forward_implications":["If the slot-based alignment is doing what the paper claims, text-image-to-video systems can be built autoregressively without the usual controllability penalty, because object identity is carried forward in slots rather than reconstructed from scratch each frame.","The reported gains mean object overlap during motion need not cause permanent object loss: the slot injection path keeps supplying the object's attributes even when occlusion makes pixels ambiguous in a particular frame.","The approach is dataset-portable in the paper's telling: training a fresh Slot Attention encoder per dataset and keeping the rest of the diffusion pipeline from scratch was enough to show gains on MNIST, CATER, and the more realistic Weizmann, KTH, and Bridge sets.","The slot count is a real design lever: the ablation shows K=5 is optimal on CATER-GEN-v2 and both too-few and too-many slots hurt the scores, so fixing the budget trades object coverage against noise.","Replacing either the SPADE fusion or the slot pathway with a simpler cross-attention option degrades the metrics, which supports the paper's conclusion that the two conditioning routes are complementary rather than redundant."],"supporting_citations":[{"why":"Supplies the Slot Attention encoder that partitions the first frame into object slots, the mechanism the paper's alignment claim depends on.","marker":"(Locatello et al. 2020)"},{"why":"Defines the TI2V task and the MNIST/CATER datasets, and provides the MAGE baseline whose reproduced numbers the paper compares against.","marker":"(Hu, Luo, and Chen 2022)"},{"why":"Provides the MAGE+ baseline and official weights used as the strongest comparison on the CATER datasets.","marker":"(Hu, Luo, and Chen 2023)"},{"why":"Provides the Seer baseline compared on CATER, including qualitative cases where Seer loses objects or generates non-existent objects.","marker":"(Gu et al. 2023)"},{"why":"Supplies SPADE scale-offset modulation, the mechanism that injects fused text-image features into the U-Net.","marker":"(Park et al. 2019)"},{"why":"Supplies Gumbel-Softmax, which lets the model adaptively select object slots with a differentiable relaxation.","marker":"(Jang, Gu, and Poole 2016)"},{"why":"Provides the DDPM denoising objective and epsilon-prediction formulation that the autoregressive video generator extends.","marker":"(Ho, Jain, and Abbeel 2020)"},{"why":"Supplies the ConvGRU temporal module that conditions each generated frame on the previous frames.","marker":"(Ballas et al. 2015)"}],"fun_headline_variants":["Slot-based text-video alignment keeps objects intact","Per-object caption alignment cuts video object loss","Decoupling objects via slots improves caption-driven motion","Object slots stop wrong-object movement in text-to-video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the grouping network pretrained to reconstruct the first frame with a fixed number of slots really does place each object in its own slot, and that the slot-text correspondence survives even when the scene holds more objects than slots, contains occlusions, or contains objects that look alike.","fun_headline_variants_meta":{"raw":{"variants":["Slot-based text-video alignment keeps objects intact","Per-object caption alignment cuts video object loss","Decoupling objects via slots improves caption-driven motion","Object slots stop wrong-object movement in text-to-video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1502,"prompt_tokens":1009,"completion_tokens":493,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":433}},"tokens_in":625,"tokens_out":493,"duration_ms":5208,"temperature":1.0,"reasoning_tokens":433,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:59:41.509460+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold the slot count at five on CATER-GEN-v2 and test captions in scenes with eight objects, then compare whether the object named in the caption is the one that actually moves; if compliance collapses once the object count passes the slot budget, the claim that slots carry object-text alignment fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Slot Attention encoder that partitions the first frame into object slots, the mechanism the paper's alignment claim depends on."}],"review_version":1}