{"id":"f2a4c22c-fada-4cdf-b5bc-0e16efbb3297","arxiv_id":"2412.18966","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Continual pre-training of a text-to-video model with block duplication plus an LLM-conditioned cross-attention improves benchmark scores, but aggregate gains hide several per-dimension regressions.","lead":"ModelGrow extends an existing text-to-video diffusion model by duplicating transformer blocks and adding a second cross-attention conditioned on a large language model, then continues pre-training on re-captioned data. The authors report improved VBench and CompBench scores over vanilla and LoRA continual training, though the gains come with several evaluation caveats.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's headline comparison confounds continual pre-training with inference-time recaptioning and CFG tuning; the LLM block may account for only a fraction of the reported gain.","rationale":"The paper's central quantitative evidence is Table 1, and the reader's conditional verdict hinges on whether the comparison is fair. I agree with the reader's weakest-assumption: the final model's inference protocol differs from the baselines on two axes that are known to affect score. The paper itself shows in Table 3 that switching the prompt from S to SL for the same model moves MLLM score from 0.6315 to 0.6737, and Figure 6 shows CFG scale materially changes quality/semantic scores; those are inference-time choices, not effects of block expansion or continual pre-training. The ablation row pair Expansion-1.4B-SL vs Expansion-1.4B-LLM-SL is a fair test of the LLM cross-attention block under a shared protocol, and it shows a real gain (0.6470 to 0.6737), so the paper is not without support. But the headline numbers compared against the base model add the prompt and CFG advantage on top of that gain, and may also reflect test-set tuning of s_llm on CompBench. I would keep the CONDITIONAL verdict: the architecture idea and same-protocol ablation are plausible enough to warrant a revision, but the main claim needs re-evaluation with matched inference protocols and ideally held-out CFG selection before it can be accepted as stated. The promised code and model release would also let the community verify the reported numbers, but the fairest single check is the baseline-with-SL rerun described above. The additional concern that no forgetting metric is reported is real but secondary; the primary defect is that the headline comparison does not isolate the continual-pretraining contribution.","tokens_in":16952,"tokens_out":6564,"duration_ms":60293,"concrete_test":"Recompute Table 1 by running the Base model, Vanilla-0.7B, LoRA-0.7B, and Expansion-1.4B with the merged short-plus-long prompt (same LLaVA-NeXT recaption and same tokenization/truncation as Expansion-1.4B-LLM-SL) and CFG st5=7, with no LLM condition, on the VBench standard prompts and T2V-CompBench prompts. Record VBench total and MLLM semantic score. If Base-SL already approaches Expansion-1.4B-LLM-SL's 78.49%/0.6737, then most of the headline gain is inference-time recaptioning; if the gap persists under this matched protocol, the continual-pretraining and LLM-block claims survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline comparison in Table 1 is not made under a single inference protocol. Section 4.1 states that the VBench standard evaluation uses the official 946 short prompts, and the baselines are run with their standard CFG. The final Expansion-1.4B-LLM model, however, is run with the merged short-plus-long prompt (Table 3, suffix SL) and with s_llm=12.5 chosen on the CompBench evaluation (Section 4.3.2, Figure 6). The reported total-score gap (78.49 vs 76.28 for the base) and the MLLM semantic gap (0.6737 vs 0.6015) therefore include an inference-time prompt-augmentation and CFG advantage that is not part of the continual-pretraining method being compared. The paper's own S-only row in Table 3 (Expansion-1.4B-LLM-S: MLLM 0.6315, subject 96.52, image 66.93) shows that with the original short prompt the LLM-conditioned model does not beat Expansion-1.4B (MLLM 0.6438, subject 96.50, image 66.52 in Table 1). The matched-protocol LLM-injection comparison (Expansion-1.4B-SL 0.6470 vs Expansion-1.4B-LLM-SL 0.6737) supports a real but much smaller effect than the headline implies. Without running all baselines under the SL prompt and the same CFG protocol, the strongest claim overstates what the continual pre-training itself contributes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies continual general pre-training of a text-to-video diffusion transformer. Starting from an Open-Sora-style DiT with 0.7B parameters, ModelGrow adds (i) a model-expansion step that duplicates transformer blocks with insert stacking and zero-initialized residual branches, and (ii) a language-understanding step that appends a gated LLM cross-attention block to every transformer block, initializes it from the T5 cross-attention weights, and feeds it template-wrapped embeddings of prompts produced by LLaVA-NeXT. At inference the short prompt and the LLM long prompt are concatenated (SL), and a two-condition classifier-free guidance (Eq. 3) is used with separate scales s_t5 and s_llm. The base model is pre-trained on a mixture of Panda-70M, Vript, WebVid, and JourneyDB; all compared methods then see the same 400k-video continual corpus with the same total step budget. Evaluation uses VBench standard prompts (946 short prompts), T2V-CompBench attribute prompts (100 prompts), and an MLLM (LLaVA-v1.6-34b) semantic score, with ablations over expansion size, stacking variant, prompt strategy, and LLM-branch components.","tokens_in":2255,"tokens_out":2346,"duration_ms":204694,"significance":"The paper is potentially useful to the T2V community: it gives a systematic comparison of three block-expansion placements (insert/prefix/suffix) and three expansion scales (1.1B/1.4B/2.1B), proposes a gated LLM cross-attention block initialized from the T5 block, and promises open-source code and models. The controlled comparison in Table 3 (Expansion-1.4B-SL vs Expansion-1.4B-LLM-SL under the same SL prompt) supports a real but modest benefit of the LLM branch (MLLM 0.6470 to 0.6737; Image Quality 65.85 to 67.28), and the SL prompt-merging idea is well motivated and validated by Figure 5 and Table 3. The two-condition CFG in Eq. (3) is a clear, easy-to-replicate formulation. The main limitation is evidential: the headline Table 1 comparisons are not apples-to-apples, and the per-dimension regressions in Appendix D are inconsistent with the paper's own broad claims, so the significance of the reported gains cannot be judged from the current evidence.","major_comments":[{"comment":"Table 1 mixes training and architecture changes with inference-time changes, so the reported advantage of Expansion-1.4B-LLM is not attributable to the continual-pretraining method alone. For the CompBench columns of Table 1, the final model is evaluated with the merged short-plus-long prompt and s_llm=12.5 (Section 4.3.2, Figure 6, Table 3), whereas no baseline is evaluated with the SL prompt or the LLM guidance term; the MLLM gain (0.6737 vs 0.6015 for the Base model and vs 0.6438 for Expansion-1.4B) therefore includes recaptioning and CFG benefits. The paper's own matched comparison (Table 3) bounds the architectural effect at 0.6470 to 0.6737 MLLM under the SL prompt, and shows that with the short prompt alone the LLM model is worse than Expansion-1.4B (MLLM 0.6315 vs 0.6438; Image Quality 66.93 vs 66.52). For the VBench standard columns, Section 4.1 prescribes the official 946 short prompts, so the remaining protocol difference is the extra s_llm=12.5 term in Eq. (3), whose contribution is not isolated because no s_llm ablation or baseline-with-matched-guidance row is reported on VBench. Please report every method under both S and SL prompting and under matched guidance, and state the protocol per row in the table caption.","section":"Table 1 vs Table 3; Section 4.3.2"},{"comment":"The LLM guidance scale is selected on the evaluation benchmark itself. Figure 6 sweeps s_llm from 4 to 20 on the CompBench prompts, and the chosen value s_llm=12.5 is then used for the reported Expansion-1.4B-LLM results, so the headline CompBench numbers are obtained with a hyperparameter tuned on the test set. The accompanying statement that scores improve as s_llm increases is also inconsistent with stopping at 12.5 rather than 20, and no plateau or optimum is shown. Furthermore, because the final configuration combines s_t5=7 with s_llm=12.5 in Eq. (3), its effective conditioning strength is roughly three times that of the baselines' single s_t5=7 term, so part of the VBench quality gain may be a guidance-strength effect. Please move the CFG selection to a validation split (or report the full curve with a justified operating point), and add a matched-guidance ablation for the baselines so the CFG contribution can be separated from the method's contribution.","section":"Figure 6 and Section 4.3.2"},{"comment":"The aggregate scores hide large per-dimension regressions of the final model relative to Expansion-1.4B on the standard VBench prompts, none of which are discussed in the main text: Dynamic Degree drops from 0.5833 to 0.3056 (Table 5); Spatial Relationship drops from 0.3330 to 0.2466, Color from 0.8360 to 0.8131, Multiple Objects from 0.3438 to 0.3229, and Overall Consistency from 0.2821 to 0.2720 (Table 6). These numbers directly qualify the main-text claims (Section 4.2) that the LLM enhancement improves quality and semantic alignment, and the Dynamic Degree collapse is severe enough to affect the usefulness of the model for motion-rich content. Because this regression may be a side effect of the large s_llm=12.5 over-guidance, the paper should include motion-aware ablations (e.g., Dynamic Degree as a function of s_llm) and explicitly discuss why the semantic gains concentrate in some dimensions while other dimensions regress.","section":"Appendix D, Tables 5 and 6; Section 4.2"}],"minor_comments":[{"comment":"The sentence \"where the w denotes the guidance scale\" follows Eq. (1), but w does not appear in Eq. (1) or its surrounding text; it looks like a leftover from a removed classifier-free-guidance equation and should be deleted or completed.","section":"Section 3.1, Eq. (1)"},{"comment":"The gating factor of the LLM cross-attention block is called lambda in the text and gamma in Figure 8; the notation should be unified, and the figure should make explicit where the tanh gate is applied.","section":"Section 3.3 vs Figure 8"},{"comment":"The tables do not state which prompt protocol (S, L, or SL) and which CFG values are used for each row; because the paper's central comparison depends on this, the captions (or a protocol table) must specify these settings per row.","section":"Table 1 and Table 3 captions"},{"comment":"The novelty claims \"the first comprehensive study of continual pre-training in text-to-video generation\" and \"we are the first work that systematically explores...\" (Section 1) should be qualified with \"to the best of our knowledge\" and checked against concurrent work on LLM-conditioned diffusion and continual T2V training, since the Related Work does not cover LLM-embedding conditioning in text-to-image diffusion beyond the gating reference to Lumina-T2X.","section":"Section 1 and Section 2.3"},{"comment":"The motivating claim that direct fine-tuning \"would lead to the model performance drop\" (Section 1) is not supported by Table 1, where Vanilla-0.7B has total score 76.27 percent versus 76.28 percent for the Base model and even improves several quality dimensions; a dedicated forgetting measurement (e.g., pre/post continual-training evaluation on held-out original-distribution prompts) would make the motivation concrete.","section":"Section 4.2, Table 1"},{"comment":"Minor presentation issues: the insert-stacking formula in Appendix A is typeset as \"P M N = x\" and should read P*M/N = x; the first-page figure caption leaks duplicated prompt strings and \"Click me\" placeholders into the text, and \"hairriding\" is missing a space; and no evaluation seeds or repeated samples are reported for the 100-prompt CompBench evaluation, whose MLLM scores are likely to be noisy.","section":"Appendix A and overall rendering"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the central risk is inference-protocol fairness rather than methodological invalidity. The matched ablation in Table 3 suggests the LLM cross-attention branch does help under the same prompt, so I do not see the comparison as hopeless; major_revision with matched-protocol recomputation seems achievable. Please confirm before accepting the final numbers that the CFG sweep in Figure 6 was not run on the evaluation prompts; test-set hyperparameter tuning would need to be disclosed prominently. I would also suggest that the authors double-check the \"first work\" novelty claims against concurrent preprints, since T2V continual pre-training and LLM-embedding conditioning are active areas."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper delivers a plausible recipe: block duplication with zero initialization lets you grow an Open-Sora-style DiT and continue training, and adding a gated LLM cross-attention block improves semantic alignment by a modest, measurable amount. The ablations on stacking variants, expansion sizes, and prompt strategies are careful, and the matched-protocol comparison in Table 3 (Expansion-1.4B-SL vs Expansion-1.4B-LLM-SL) is the right way to isolate the architecture's effect. The novelty is limited—block duplication is LLaMA Pro, zero-init gating is Lumina-T2X, recaptioning is DALL-E 3—but applying these to T2V and studying the design choices fills a real gap in the literature.\n\nThe soft spots are in the evaluation. The headline Table 1 is not apples-to-apples. The final model uses the merged short+long prompt and s_llm=12.5 selected on the CompBench benchmark, while baselines run with short prompts and their default CFG. That confound alone can explain a large part of the reported gap. The paper's own Table 3 shows that with the original short prompt, Expansion-1.4B-LLM-S gets MLLM 0.6315, below Expansion-1.4B's 0.6438, and subject scores are essentially tied. So the LLM cross-attention's real contribution is the SL-matched gap (MLLM 0.6470 to 0.6737), which is meaningful but much smaller than the headline implies.\n\nTwo more issues. The catastrophic-forgetting claim is never tested—no evaluation on the original pretraining distribution is reported. And the appendix shows a large regression on Dynamic Degree for Expansion-1.4B-LLM (0.3056 vs 0.5833 for Expansion-1.4B) with no discussion. That's a quality cost hiding in the aggregate.\n\nOverall, this is a credible engineering contribution with fixable evaluation flaws. It deserves a serious referee, not a desk reject. The referee should push for a fair protocol (baselines with the same prompt augmentation and CFG), a forgetting metric, and release of code and model. The method itself likely works; the evidence as presented overstates its gains.\n\nRecommended: send to peer review, with expectations of revision.","headline":"A credible engineering recipe for growing T2V models, but the headline gains are inflated by inference-time recaptioning and CFG tuning, and the forgetting claim is untested.","tokens_in":17854,"tokens_out":4558,"would_cite":true,"duration_ms":38616,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that continual pre-training can grow a text-to-video model by duplicating transformer blocks and adding an LLM-conditioned cross-attention block, beating vanilla fine-tuning and LoRA on quality and semantic alignment.","keywords":["text-to-video generation","continual pre-training","model expansion","block duplication","large language models","diffusion transformer","semantic alignment","prompt re-captioning"],"falsifier":"Run the final Expansion-1.4B-LLM model on VBench standard prompts using only the short prompt and the base CFG scale, with no LLM condition, and compare it with the base model and LoRA baseline under identical conditions; if the semantic-score gap shrinks to near zero, the claimed advantage is largely an artifact of recaptioning and guidance tuning. Alternatively, evaluate a vanilla fine-tuned model with the same short-plus-long prompt and tuned CFG; if it matches the expansion model, block duplication is not the driver.","tokens_in":16712,"feed_emoji":"🎬","tokens_out":5737,"duration_ms":49772,"temperature":0.7,"pith_summary":"The paper argues that a pre-trained text-to-video diffusion model can be improved through continual pre-training rather than retraining from scratch, and that the two essential ingredients are structural growth and better language conditioning. It shows that duplicating transformer blocks, with zero-initialized copies that act as identity mappings, lets the model absorb new data without catastrophic forgetting. It then adds a second cross-attention block conditioned on frozen large language model embeddings, giving the model a controllable way to follow long, detailed prompts. On its benchmarks, the grown 1.4B model with LLM conditioning outscores the base model, a same-size vanilla fine-tune, and a LoRA fine-tune on both overall video quality and semantic alignment. The claim is that these gains come from the continual pre-training recipe itself, not from extra data or larger compute.","feed_headline":"Growing a video model beats retraining it, hitting 78.49%","feed_subtitle":"Block duplication plus an LLM text condition beats vanilla fine-tuning and LoRA on video quality and prompt alignment.","key_machinery":"The load-bearing mechanism is the zero-initialized duplicated block. Expanding the transformer by copying adjacent blocks and zeroing the last linear layer of every residual sub-block makes each new block an identity at initialization, so the loss does not spike and old capabilities are not disrupted; training then lets the new blocks specialize to new data. The second mechanism is the LLM cross-attention block: a second cross-attention module in each transformer block, initialized from the T5 cross-attention weights, gated by tanh with a zero-initialized gate, and conditioned on embeddings from a frozen LLM. Together they let the model grow in capacity and in its ability to follow long prompts while keeping the original text-conditioning path intact.","core_discovery":"The central discovery is that continual pre-training of a pre-trained text-to-video transformer can be decomposed into model expansion and language-understanding enhancement. Model expansion duplicates transformer blocks and interleaves the copies with the originals, zero-initializing the new blocks so they behave as identity mappings at the start of training, which preserves previously learned behavior while adding capacity. Language understanding is enhanced by appending an LLM-conditioned cross-attention block after the existing T5 cross-attention block in every transformer block, initializing it from the T5 cross-attention weights and gating it with a zero-initialized tanh gate, and by feeding the model a concatenation of the original short prompt and an LLM-expanded long prompt. With these two mechanisms, the resulting Expansion-1.4B-LLM model improves over the base model and over vanilla and LoRA continual fine-tuning on VBench and CompBench, with the largest gains in semantic alignment.","pith_inferences":["A reader should not take the headline VBench gap as purely due to the training method: the final model was evaluated with the merged short-plus-long prompt and a separately tuned LLM guidance scale of 12.5, while baselines used short prompts and the original guidance, so inference recipe differences are confounded with the method.","The method is demonstrated on a single base architecture and dataset budget; whether block expansion transfers to other text-to-video backbones or to higher resolutions and longer videos is untested, though nothing in the mechanism obviously prevents it.","The LLM-conditioned cross-attention block effectively lets a frozen LLM act as a second, more semantically rich text encoder, so one testable consequence is that stronger LLM text encoders should further improve semantic alignment without retraining the video model.","The zero-initialized identity blocks suggest an analogy to residual learning, where each new block learns a correction on top of the pretrained stack; this could make the method more stable than naive fine-tuning at larger expansion ratios."],"forward_implications":["Continual pre-training can serve as a cheaper alternative to retraining a text-to-video model from scratch, since the base model is reused and only the new and expanded parameters are trained.","Model expansion via zero-initialized block duplication mitigates catastrophic forgetting relative to full fine-tuning at the same parameter count, because the identity initialization preserves the pretrained trajectory.","Adding an LLM-conditioned cross-attention block gives a controllable second text condition, whose influence can be tuned at inference through a separate classifier-free guidance scale.","Concatenating the original prompt with a long recaption preserves key objects from the user prompt while adding detail, a simple inference change that improves semantic fidelity.","The recipe transfers to other transformer-based text-to-video models in principle, since it only requires duplicating blocks and adding a cross-attention module."],"supporting_citations":[{"why":"Supplies the block-expansion idea that the paper adapts from language models to text-to-video generation.","marker":"[51]"},{"why":"Provides the Open-Sora V1.0 base architecture that the paper continually pre-trains.","marker":"[56]"},{"why":"Defines the LoRA continual fine-tuning approach used as the parameter-efficient baseline.","marker":"[28]"},{"why":"Introduces recaptioning, which the paper modifies by concatenating short and long prompts.","marker":"[3]"},{"why":"Generates the long training prompts and serves as the multimodal evaluator for the semantic score.","marker":"[33]"},{"why":"Provides the VBench evaluation suite used for quality and semantic scores.","marker":"[29]"},{"why":"Supplies the T2V-CompBench consistency attribute prompts used to evaluate LLM enhancement.","marker":"[43]"},{"why":"Introduces gated zero-initialization, which the paper borrows for the LLM cross-attention block.","marker":"[16]"},{"why":"Provides the two-condition classifier-free guidance formula used at inference.","marker":"[5]"},{"why":"Defines the DiT transformer block structure that the paper expands.","marker":"[40]"}],"fun_headline_variants":["ModelGrow boosts video AI by expanding blocks and LLM prompts","Continual T2V pre-training: expand model and add LLM text","Model expansion and LLM encoders improve video generation","Duplicate blocks, add LLM: ModelGrow grows video models","Video model grows skills via block expansion and LLM guidance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes all methods are evaluated under the same inference protocol, but the reported best model uses the merged short-and-long prompt and a tuned LLM guidance scale while baselines use the short prompt and the original scale.","fun_headline_variants_meta":{"raw":{"variants":["ModelGrow boosts video AI by expanding blocks and LLM prompts","Continual T2V pre-training: expand model and add LLM text","Model expansion and LLM encoders improve video generation","Duplicate blocks, add LLM: ModelGrow grows video models","Video model grows skills via block expansion and LLM guidance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1455,"prompt_tokens":978,"completion_tokens":477,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":403}},"tokens_in":594,"tokens_out":477,"duration_ms":5320,"temperature":1.0,"reasoning_tokens":403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:51.183395+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the final Expansion-1.4B-LLM model on VBench standard prompts using only the short prompt and the base CFG scale, with no LLM condition, and compare it with the base model and LoRA baseline under identical conditions; if the semantic-score gap shrinks to near zero, the claimed advantage is largely an artifact of recaptioning and guidance tuning. Alternatively, evaluate a vanilla fine-tuned model with the same short-plus-long prompt and tuned CFG; if it matches the expansion model, block duplication is not the driver.","supporting_citations":[{"cited_title":"LLaMA pro: Progressive LLaMA with block expansion","cited_arxiv_id":null,"evidence_quote":"Supplies the block-expansion idea that the paper adapts from language models to text-to-video generation."},{"cited_title":"Open-sora: Democratizing efficient video production for all, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the Open-Sora V1.0 base architecture that the paper continually pre-trains."},{"cited_title":"Lora: Low- rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"Defines the LoRA continual fine-tuning approach used as the parameter-efficient baseline."},{"cited_title":"Improving image generation with better captions","cited_arxiv_id":null,"evidence_quote":"Introduces recaptioning, which the paper modifies by concatenating short and long prompts."},{"cited_title":"VBench: Com- prehensive benchmark suite for video generative models","cited_arxiv_id":null,"evidence_quote":"Provides the VBench evaluation suite used for quality and semantic scores."}],"review_version":1}