{"id":"ca423ff1-0c62-449c-b2e2-1ffe1cb63f28","arxiv_id":"2411.17470","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":14,"one_line_summary":"Video diffusion transformers follow scaling laws, and tuning learning rate and batch size per model and data size makes those laws precise enough to predict loss and optimal model size at larger scales.","lead":"Video diffusion model performance follows predictable power-law scaling once learning rate and batch size are tuned for each model size and data budget. The authors fit these relationships on small models, then show the predictions hold on a 0.72B-parameter model trained on 140B tokens, pointing toward cheaper large-scale video AI training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (17) drops the α_T exponent when substituting C = C_token T into L(T,N); since Eq. (18) and the 3.57% slope agreement are derived from it, the predicted-Nopt comparison and the 40.1% inference-cost claim may rest on an algebraic error.","rationale":"The paper's strongest evidence is the out-of-sample loss prediction at 0.72B/140B and 1.07B/10B; those are genuine empirical tests and should be credited. The extrapolation concern raised by the reader is real, especially because the paper's own limitation section says low-resolution small models may not transfer. However, I see a more pointed, immediately checkable problem: the printed derivation that connects the validated loss law to the predicted optimal model size contains a dropped exponent. Because the 39.9%/40.1% parameter/inference saving and the Fig. 7 slope comparison are the quantitative payoff of the paper, this algebraic step is load-bearing. If the re-derivation reproduces Eq. (18), the issue is a typo and the claims stand; if not, the paper's strongest applied claim loses its support. In either case the loss-law prediction claim is separable and remains credible, which is why I would not reject the paper outright. The verdict should stay conditional, with the re-derivation as an explicit required check.","tokens_in":45,"tokens_out":24505,"duration_ms":241282,"concrete_test":"Independently re-derive Eq. (17) from Eq. (15) and C = Ctoken T, with Ctoken/N = (3/4)(7+nctx/d) and d = (8N)^{1/3} (or the exact architecture table in Appendix D), restoring the α_T exponent. Numerically minimize L(N) for C in {3e17, 6e17, 1e18, 3e18, 6e18}, refit a power law for Nopt(C), and compare its exponent and constant with Eq. (18) and the empirical Eq. (14). If the refitted exponent differs from 0.4294 by more than about 0.03, or the slope deviation in Fig. 7 exceeds a few percent, the predicted-Nopt validation and the 40.1% inference-cost claim are not supported by the current derivation. If the refitted curve reproduces Eq. (18), the printed error is typographical and the claims stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the derivation of the predicted optimal model size in Sec. 4.3.2. Eq. (15) is L(T,N) = (Tc/T)^αT + (Nc/N)^αN + L∞. Substituting C = Ctoken T with Ctoken/N = (3/4)(7 + nctx/d) yields (Tc * 3N(7+nctx/d)/(4C))^αT for the first term, not Tc * 3N(7+nctx/d)/(4C) as printed in Eq. (17). The exponent αT = 0.2917 has been dropped. This is not cosmetic: minimizing the printed linear form gives Nopt ∝ C^{1/(1+αN)} ≈ C^{0.758}, while restoring the exponent (and treating the dimensionless ratio as fixed) gives Nopt ∝ C^{αT/(αT+αN)} ≈ C^{0.478}. Neither matches the stated hat-Nopt = 0.8705 C^{0.4294}, so Eq. (18) cannot be obtained from the printed derivation. Moreover, because d scales as N^{1/3} under d/nlayer = 128 and nctx is fixed, the factor (7 + nctx/d) is not constant; the minimization is not exactly power-law, which needs to be stated and handled. Eq. (18) and the 3.57% slope-deviation comparison in Fig. 7 are the quantitative support for the paper's claim that optimal hyperparameters yield more precise validation-loss predictions, and the 39.9%/40.1% parameter/inference saving is derived from the same Nopt comparison. If the algebra is wrong, those headline numbers are unsupported even though the two out-of-sample loss predictions (0.72B/140B, 1.07B/10B) remain valid evidence for Eq. (15).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies scaling laws for video diffusion transformers (Cross-DiT architecture). It proposes two power-law fits for the optimal batch size and learning rate as functions of model size N and training tokens T, and a three-term validation-loss law L(T,N) = (Tc/T)^α_T + (Nc/N)^α_N + L∞. The authors claim that training with these optimal hyperparameters yields more precise loss fits, allows prediction of optimal model size versus compute budget, and leads to a 39.9% parameter saving and a 40.1% inference-cost reduction at a 1e10 TFlops budget. Out-of-sample checks are reported at 1.07B parameters / 10B tokens and 0.72B parameters / 140B tokens (also described as 5.85e20 FLOPs), with deviations of 0.03% and 0.15%.","tokens_in":19485,"tokens_out":11028,"duration_ms":94036,"significance":"If the proposed laws hold, the paper would provide practically useful guidance for allocating compute between model size and data in video diffusion training, and the explicit treatment of batch size and learning rate is a useful step beyond standard LLM-style scaling laws. The strongest evidence is the out-of-sample validation of L(T,N) at 1.07B/10B and the claimed 0.72B point, which go beyond internal curve fitting. The paper also gives a detailed FLOPs-per-token accounting and honestly states limitations (constant learning rate, low resolution, small models, validation-loss-only evaluation). However, the algebraic derivation of the predicted optimal model size contains a serious error, and one flagship validation point is described inconsistently in terms of tokens versus compute, so the headline quantitative claims need to be re-derived and re-stated before the paper can be accepted.","major_comments":[{"comment":"Equation (17) drops the exponent α_T when substituting C = C_token T into Eq. (15). The correct substitution is L(N) = (Tc · 3N(7+nctx/d)/(4C))^{α_T} + (Nc/N)^{α_N} + L∞, not the printed linear first term Tc · 3N(7+nctx/d)/(4C). This is not cosmetic: minimizing the printed form gives Nopt ∝ C^{1/(1+α_N)} ≈ C^{0.758}, while restoring the exponent and holding (7+nctx/d) fixed gives Nopt ∝ C^{α_T/(α_T+α_N)} ≈ C^{0.478}; neither equals the stated Eq. (18), Nopt = 0.8705·C^{0.4294}. The derivation of Eq. (18), the 3.57% slope agreement in Fig. 7, and the associated parameter/inference saving estimates therefore rest on an algebraic error. Please re-derive Eq. (18) or state explicitly how it was obtained (e.g., as a numerical power-law fit to the corrected minimization, including the d(N) dependence from d/nlayer = 128).","section":"Sec. 4.3.2, Eq. (17)"},{"comment":"The 0.72B out-of-sample validation point is internally inconsistent. The text says this model was trained on 140B tokens, while Sec. 4.3.1 and Fig. 5b assign the same point a compute budget of 5.85e20 FLOPs. Using the paper's own compute model from Appendix D (Table 6), a 719.3M-parameter model with d=1792 and nctx=1,113,632 has C_token ≈ (3/4)(7+nctx/d)N ≈ 3.4e11 FLOPs per token. That gives 140B tokens ≈ 4.8e22 FLOPs, whereas 5.85e20 FLOPs corresponds to about 1.7B tokens. These two descriptions cannot both be right, and the 0.15% deviation claim depends on which run was actually performed. Please reconcile the token count, the FLOPs budget, and the figure labels.","section":"Sec. 4.2 and Fig. 5b vs. Sec. 4.3.1"},{"comment":"The headline parameter-saving and inference-cost numbers are obtained by extrapolating the Nopt-vs-C scaling laws to C = 1e10 TFlops = 1e22 FLOPs, which is more than three orders of magnitude beyond the largest fitted compute budget (6e18 FLOPs). The 0.72B run at 5.85e20 FLOPs provides partial support, but no experiment at 1e22 FLOPs validates the slope of the extrapolated Nopt(C) law. In addition, the 3.57% agreement in Fig. 7 compares the predicted Nopt(C) from Eq. (18) with the empirical Nopt(C) from Eq. (14) on the same five IsoFLOP budgets used to fit both curves; this is an internal-consistency check, not an out-of-sample validation. The paper should present the 1e22 extrapolation as a prediction and state the untested range explicitly, or add an intermediate out-of-sample compute budget.","section":"Sec. 4.3.1 and Abstract (40.1% claim)"}],"minor_comments":[{"comment":"Equation (23) appears dimensionally inconsistent: from Eq. (22) and C = C_token T, the correct expression is Topt = (4/(3(7+nctx/d)·0.8705))·C^{0.5706}, not (4/3)(7+nctx/d)·C^{0.5706} as printed.","section":"Supplementary Eq. (23)"},{"comment":"The fitted coefficients for β_B, γ_B, α_η, etc. are presented without confidence intervals or sensitivity to the 0.02% near-optimal threshold used to define 'optimal' hyperparameters; adding this would help readers judge the stability of the fitted exponents.","section":"Sec. 3.4 / Tables 1 and 2"},{"comment":"There are several typos and missing spaces in figure labels and text, e.g., 'Traing T okens' in Fig. 1 and Fig. 5, 'FixedSuboptimalHyperparametersOptimalHyperparameters' in Fig. 7, and 'video-specfic' in Sec. 5; these should be corrected.","section":"Figures and text"},{"comment":"The sentence 'which can also make accurate predictions for image generation' is ungrammatical and should be rewritten, e.g., 'showing that the approach also gives accurate predictions for image generation.'","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper has a valuable empirical core, particularly the out-of-sample loss predictions, but the algebraic error in Eq. (17) and the inconsistency in the 0.72B validation description need to be fixed before the quantitative claims can be trusted. The reported 40.1% inference-cost reduction should be presented as an extrapolated prediction unless additional validation is provided. I believe the central scaling-law story is defensible and within the scope of the journal, but the current draft is not publishable as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the short version: the paper's empirical core is real and worth reading, but there is a load-bearing algebraic error in Section 4.3.2. Equation (17) drops the α_T exponent when substituting C = C_token T into L(T,N); the first term should be raised to the 0.2917 power. With the printed linear form, minimizing gives N ∝ C^{0.758}; with the exponent restored (and the nctx/d factor treated as fixed), N ∝ C^{0.478}. Neither yields the stated hat-Nopt = 0.8705 C^{0.4294}. So the 3.57% slope-deviation comparison and the 40.1% inference-cost claim are not supported by the derivation as printed.\n\nWhat is actually new and good: the paper establishes scaling laws for video diffusion transformers, with optimal batch size and learning rate modeled as power laws in N and T, and validates the loss law L(T,N) out-of-sample at 1.07B/10B tokens (0.03% error) and 0.72B/140B tokens (0.15% error). Those two points are strong evidence that the fitted loss surface captures real structure; they do not depend on the broken Nopt derivation. The paper also checks the hyperparameter laws themselves with a grid search at 1.07B, and provides an image-generation comparison in the appendix. That is more than most scaling-law papers do.\n\nSoft spots beyond the algebra: no error bars on any fitted exponent; the 'optimal hyperparameter' data points are chosen by a 0.02% loss threshold, which is arbitrary; the claim that video DiTs are more hyperparameter-sensitive than LMs is asserted but not quantified; and the headline compute-budget extrapolation goes to 1e22 FLOPs, far beyond the fitted range. These are secondary. The primary issue is that Section 4.3.2, which is the quantitative support for the efficiency claim, does not hold together as written. If the authors provide the actual minimization (likely numerical, over discrete layers), the stated exponent may be recoverable, but the reader cannot verify it now.\n\nMy view: this deserves a serious referee. The empirical core is valuable and reproducible enough to be worth engaging; the derivation error is fixable and should be caught in review. I would not cite the 40.1% savings number until it is corrected. Bring it to the reading group if you want a case study in how a small typo can invalidate a headline claim.","headline":"Solid empirical scaling-law study for video DiTs, but Eq 17 drops the α_T exponent and the derived Nopt comparison and 40.1% savings claim are unsupported as printed.","tokens_in":20025,"tokens_out":3700,"would_cite":true,"duration_ms":31705,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that video diffusion transformers obey precise scaling laws for validation loss, batch size, and learning rate, with fitted power laws that predict held-out losses within 0.15 percent.","keywords":["scaling laws","video diffusion transformers","optimal hyperparameters","learning rate","batch size","validation loss","compute-optimal model size","power law"],"falsifier":"Train a video diffusion transformer far outside the fitted region, for example roughly 1.5B parameters on roughly 200B tokens, using the batch size and learning rate from the fitted laws, and compare the measured validation loss against the $L(T,N)$ prediction; likewise, measure the optimal model size by IsoFLOP curves at $C=10^{22}$ FLOPs and check whether $N_{\\mathrm{opt}}$ still follows the same exponent. A deviation beyond the roughly 0.15% seen in the paper would mean the power laws bend.","tokens_in":18742,"feed_emoji":"🎬","tokens_out":6974,"duration_ms":54983,"temperature":0.7,"pith_summary":"Video diffusion transformers obey scaling laws, but only when the batch size and learning rate are also scaled optimally rather than held fixed. The paper fits power laws $B_{\\mathrm{opt}}=\\alpha_B T^{\\beta_B}N^{\\gamma_B}$ and $\\eta_{\\mathrm{opt}}=\\alpha_\\eta T^{\\beta_\\eta}N^{\\gamma_\\eta}$ for those hyperparameters across model size $N$ and training tokens $T$, then fits the validation loss as $L(T,N)=(T_c/T)^{\\alpha_T}+(N_c/N)^{\\alpha_N}+L_\\infty$. With these choices the loss law predicts held-out runs closely: a 1.07B-parameter model trained on 10B tokens comes within 0.03% of prediction and a 0.72B model trained on 140B tokens within 0.15%. If the laws hold beyond the fitted range, practitioners can pick the model size and hyperparameters before large training runs, and can deliberately choose a smaller model near the flat loss minimum, cutting inference cost by about 40.1% at a compute budget of $10^{10}$ TFlops with only minor loss degradation.","feed_headline":"Fitted power laws forecast video DiT loss within 0.15%","feed_subtitle":"Optimal batch size and learning rate make diffusion transformer scaling predictable, cutting inference cost by about 40%.","key_machinery":"The central mechanism is a stepwise-loss analysis of mini-batch SGD. The expected loss change at step $k$ is approximated as $\\Delta L_k \\approx -\\eta\\|\\nabla L\\|^2 + \\tfrac12\\eta^2(\\nabla L^\\top H \\nabla L + \\mathrm{tr}(H\\Sigma_k)/B)$, where $\\mathrm{tr}(H\\Sigma_k)/B$ is the gradient-noise term that shrinks with batch size. Balancing per-step gain against the number of steps $T/B$ yields power-law forms $B_{\\mathrm{opt}}=\\alpha_B T^{\\beta_B}N^{\\gamma_B}$ and $\\eta_{\\mathrm{opt}}=\\alpha_\\eta T^{\\beta_\\eta}N^{\\gamma_\\eta}$; the loss law $L(T,N)=(T_c/T)^{\\alpha_T}+(N_c/N)^{\\alpha_N}+L_\\infty$ then separates the data-limited error, the model-limited error, and an irreducible entropy floor $L_\\infty$. These fitted laws are what carry the extrapolation to larger models and compute budgets.","core_discovery":"On its own terms, the paper establishes that the validation loss of a video diffusion transformer is a predictable power-law function of model size and training tokens, provided each run uses the optimal batch size and learning rate for that $(N,T)$. It derives these optima from a mini-batch SGD analysis in which the per-step loss improvement is $-\\eta\\|\\nabla L\\|^2 + \\tfrac12\\eta^2(\\nabla L^\\top H \\nabla L + \\mathrm{tr}(H\\Sigma)/B)$; larger models have larger Lipschitz constants and need smaller learning rates and larger batches, while more tokens shift the balance toward larger batches. The fitted exponents are $\\beta_B=0.8080$, $\\gamma_B=0.1906$, $\\beta_\\eta=-0.0453$, $\\gamma_\\eta=-0.1619$, and the loss law has $T_c=0.0373$, $\\alpha_T=0.2917$, $N_c=0.0082$, $\\alpha_N=0.3188$, $L_\\infty=0.4856$. The paper validates the loss law by training a 1.07B model on 10B tokens (0.03% deviation) and a 0.72B model on 140B tokens (0.15% deviation), and shows that fixed suboptimal hyperparameters overestimate the optimal model size, with a 30.26% slope deviation versus 3.57% when hyperparameters are optimal.","pith_inferences":["The flat-loss-near-optimum structure implies the fitted exponents only need to be approximately right for the practical decisions on model size and hyperparameters to remain near-optimal; a direct test would be to perturb $B$ and $\\eta$ by the same relative amount at a new scale and measure the loss difference.","The same methodology should be re-fit, not reused, when resolution, frame count, architecture, or optimizer and scheduler change; the paper itself notes its laws were fitted at 256x256 resolution, 17 frames, and constant learning rate, so the exponents are likely setup-specific.","A natural extension is to treat the learning-rate schedule as another hyperparameter axis; the convergence-bound argument suggests an optimal schedule could be folded into the $\\eta_{\\mathrm{opt}}$ law rather than requiring a separate constant-rate assumption.","If the approach transfers to other architectures or modalities, the 40.1% inference saving is not a fixed property of video DiT but a consequence of where the conventional rule lands on the flat optimum; comparing the two rules at several compute budgets would show how the saving changes with budget."],"forward_implications":["Given a compute budget, the fitted law $N_{\\mathrm{opt}}=0.8705\\,C^{0.4294}$ selects the model size before training; following it at $10^{10}$ TFlops saves about 39.9% of parameters relative to the fixed-suboptimal rule while keeping validation loss nearly unchanged.","For any model size and token budget, validation loss is predictable through the $L(T,N)$ formula, so non-optimal sizes chosen for inference cost or hardware constraints can still have their loss anticipated.","The optimal batch size and learning rate for a target scale can be read off the two hyperparameter laws, removing the need for a grid search at large scale.","Because the loss curve is flat near the optimum, choosing a smaller model than the nominal optimum trades only a small, predictable loss increase for a large inference-cost reduction, which is the origin of the claimed 40.1% saving.","Fixed-suboptimal hyperparameters, by contrast, overestimate the optimal model size and make the loss fit substantially worse (MSE $4.31\\times10^{-7}$ vs $2.35\\times10^{-7}$), so hyperparameter choice is part of the scaling law, not an afterthought."],"supporting_citations":[{"why":"Supplies the power-law form for optimal batch size and learning rate, and the compute-optimal allocation framework that the paper extends by making model size and data size explicit variables.","marker":"[1]"},{"why":"Supplies the loss decomposition $L(T,N)=(T_c/T)^{\\alpha_T}+(N_c/N)^{\\alpha_N}+L_\\infty$ and the IsoFLOP methodology for identifying optimal model size.","marker":"[26]"},{"why":"Supplies the gradient-noise and large-batch training analysis that underpins the derivation of the optimal batch size law.","marker":"[38]"},{"why":"Provides the baseline OpenAI scaling-law view that smaller batches are more compute-efficient, which the paper argues fails for video diffusion transformers.","marker":"[29]"},{"why":"Supplies the Cross-DiT architecture and pretrained initialization used in all the paper's experiments.","marker":"[7]"},{"why":"Supplies the Panda-70M training dataset and the validation subset on which all loss measurements are made.","marker":"[9]"},{"why":"Supplies the diffusion transformer backbone that the Cross-DiT architecture builds on and that defines the model family under study.","marker":"[41]"},{"why":"Supports the use of validation loss as a reliable proxy for generation performance, which is the evaluation choice the paper relies on.","marker":"[13]"}],"fun_headline_variants":["Video DiT scaling: optimal batch and LR yield 0.15% loss fit","Optimal hyperparameters make video DiT scaling laws precise","Precise video DiT scaling laws: optimal batch & LR cut inference 40%","Scaling laws for video DiTs: optimal hyperparameters slash inference cost","Video diffusion transformers: precise scaling via optimal batch and LR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the three power laws fitted on models up to 0.26B parameters and 12B tokens keep their shape all the way to 0.72B parameters with 140B tokens and to $10^{22}$ FLOPs; if the exponents bend outside the fitted region, the predicted losses, model sizes, and the 40.1% saving would not hold.","fun_headline_variants_meta":{"raw":{"variants":["Video DiT scaling: optimal batch and LR yield 0.15% loss fit","Optimal hyperparameters make video DiT scaling laws precise","Precise video DiT scaling laws: optimal batch & LR cut inference 40%","Scaling laws for video DiTs: optimal hyperparameters slash inference cost","Video diffusion transformers: precise scaling via optimal batch and LR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001125,"raw_usage":{"total_tokens":4729,"prompt_tokens":1046,"completion_tokens":3683,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":3586}},"tokens_in":662,"tokens_out":3683,"duration_ms":23056,"temperature":1.0,"reasoning_tokens":3586,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:50:33.780773+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a video diffusion transformer far outside the fitted region, for example roughly 1.5B parameters on roughly 200B tokens, using the batch size and learning rate from the fitted laws, and compare the measured validation loss against the $L(T,N)$ prediction; likewise, measure the optimal model size by IsoFLOP curves at $C=10^{22}$ FLOPs and check whether $N_{\\mathrm{opt}}$ still follows the same exponent. A deviation beyond the roughly 0.15% seen in the paper would mean the power laws bend.","supporting_citations":[{"cited_title":"Panda-70m: Captioning 70m videos with multiple cross-modality teachers","cited_arxiv_id":null,"evidence_quote":"Supplies the Panda-70M training dataset and the validation subset on which all loss measurements are made."}],"review_version":1}