{"id":"a950acd1-db89-4ff6-a109-bd78568e2acc","arxiv_id":"2506.13497","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DDiT cuts text-to-video serving latency by up to 1.44x via DiT-VAE phase decoupling and step-level, starvation-aware GPU reassignment.","lead":"DDiT is a serving system for text-to-video models that reassigns GPUs at each denoising step and separates the diffusion transformer from the video decoder, reporting up to 30.4 percent lower p99 latency than fixed-deployment baselines. A generalist reader may care because efficient video-generation serving directly affects the cost and latency of deploying AI video models at scale.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The no-batching premise (Insight 1) is contradicted by the paper's own Figure 17: at 144p, batch size 2 nearly doubles throughput, so DDiT's FCFS design and baselines omit a plausible batching-aware competitor.","rationale":"The reader's weakest_assumption identified Insight 1 and the missing batching baseline as the most load-bearing issue; this stress-test agrees. The additional evidence that makes the concern concrete is the paper's own Figure 17, which numerically contradicts the assertion that batching cannot improve throughput at low/medium resolutions. Since DDiT's scheduler is FCFS and sequential by design, and all baselines also exclude batching, the central claim of significant latency improvement is compared against a restricted baseline set. The concrete test would settle whether this concern lands: if a batching-aware baseline closes the gap, the paper's central claim is overstated; if batching does not help in end-to-end serving, Insight 1 stands and the comparison is acceptable on this axis. I also considered the multi-node emulation and the idealized theoretical optimum, but they are secondary because the no-batching premise undermines the design itself and the baseline selection. The internal contradiction in Section 3.1 is a correctness risk, not merely a lack of experimental breadth. The verdict remains conditional, matching the reader: the paper should be accepted only if the batching question is resolved and the evaluation is made reproducible.","tokens_in":21646,"tokens_out":5354,"duration_ms":50693,"concrete_test":"Recompute throughput = batch_size / latency from the Figure 17 DiT table for 144p and 240p at parallel size 1; if throughput increases by more than 10% from batch 1 to batch 2, then implement a continuous-batching variant of VideoSys that batches same-resolution requests with memory-aware limits and rerun the Figure 10 single-node workloads (same arrival rates and resolution mixes) against DDiT. If that batching baseline matches or beats DDiT on p99 and average latency, or if DDiT's claimed margins reduce materially, Insight 1 and the headline comparison need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DDiT's headline latency gains rest on Insight 1 (Section 3.1): 'the optimal scheduling policy in T2V is to process requests sequentially, one at each time, because the GPU computational capacity is always limited.' This premise justifies the FCFS, non-batching scheduler (Section 4.2.3) and the absence of any batching-capable baseline in Figures 10-12. However, the same section and Figure 17 contain a direct internal contradiction. The text observes that 'the throughput scales linearly with the batch size at smaller resolutions' while concluding that 'Batching can not improve the throughput of the T2V system.' The Figure 17 data for DiT with parallel size 1 show 144p latencies of 3.44s (batch 1), 3.67s (batch 2), 6.49s (batch 4), and 12.22s (batch 8), i.e., per-request throughput rising from 0.29 to 0.55, 0.62, and 0.65 req/s; 240p also rises by about 16% from batch 1 to batch 8. This does not indicate saturation at batch size 1. Consequently, the claim that batching cannot improve throughput is not supported by the paper's own measurements, and the comparison set excludes a batching-aware system, which is the standard technique in LLM serving (e.g., continuous batching). If batching improves throughput for the low/medium resolutions that dominate real T2V workloads, then sequential FCFS is not obviously optimal, and DDiT's 1.44x and 30% latency advantages may shrink or disappear when measured against a batching-aware baseline. This concern is internal to the paper, not a disagreement with external consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DDiT, a serving system for text-to-video (T2V) diffusion transformers. DDiT decouples the DiT and VAE phases into independent elastic units, determines an optimal degree of sequence parallelism (DoP) per resolution from offline profiling, and schedules DiT execution at single-step granularity using a greedy algorithm that promotes the DoP of 'hungry' requests based on a starvation-time priority. The evaluation, using OpenSora models and VideoSys-derived baselines, reports that DDiT outperforms the baselines by up to 1.44x in p99 latency and 1.43x in average latency (single-node and multi-node, the latter being an emulation), with a breakdown of the DiT-VAE decoupling and DoP promotion mechanisms, plus a comparison against a theoretical optimal scheduling algorithm.","tokens_in":22003,"tokens_out":4210,"duration_ms":43631,"significance":"If the empirical results hold, the paper makes a useful systems contribution to T2V serving. The strengths are: (1) the observation that DiT and VAE have distinct parallelism behaviors and that a single static DoP is suboptimal across mixed resolutions and arrival rates; (2) the design of step-level DoP promotion with measured overhead below 1 ms (Figure 15); and (3) a broad ablation split (DiT-VAE decoupling, DoP promotion) that isolates the benefit of each mechanism. The paper also ships reproducible pseudocode for its scheduling algorithms. However, the headline claims rest on a load-bearing premise about batching that is internally contradicted by the paper's own experimental data, and the multi-node evaluation is an emulation on a single server. These issues materially affect whether the claimed gains would persist against a more complete set of serving baselines.","major_comments":[{"comment":"Insight 1, which motivates the FCFS no-batching scheduler (Section 4.2.3) and the absence of batching-aware baselines in Figures 10-12, is not supported by the paper's own measurements. The text states that 'the throughput scales linearly with the batch size at smaller resolutions' and then concludes 'Batching can not improve the throughput of the T2V system.' Figure 17 directly contradicts this: at 144p with parallel size 1, DiT latency is 3.44 s for batch 1 and 3.67 s for batch 2, i.e., per-request throughput rises from about 0.29 req/s to 0.55 req/s; at 240p the throughput also increases with batch size. If batching improves throughput for low and medium resolutions, then sequential FCFS is not obviously optimal, and the comparison set omits a plausible batching-aware competitor. Please either revise Insight 1 to reflect the measured superlinear throughput at small batch sizes, or add a batching-capable baseline (e.g., continuous batching) to the end-to-end evaluation to show that the headline latency gains are not an artifact of excluding such a system.","section":"Section 3.1 and Figure 17"},{"comment":"The multi-node experiments, which are the basis for the claims of at least 30.4% p99 and 30% average latency reduction, are described as an emulation of an 8-node, 64-GPU cluster on a single server with eight H800 GPUs interconnected via NVLink. The manuscript does not explain how the 200 Gbps RDMA inter-node network, cross-node communication overhead, and multi-node topology are emulated. Without this detail, the multi-node results in Figures 11 and 12 cannot be interpreted as evidence of scalability; they may only reflect single-node behavior with artificial partitioning. Please describe the emulation methodology precisely or report results on a real multi-node cluster.","section":"Section 6.1 'Testbed' and Section 6.2 'Multi-Node Performance'"},{"comment":"All end-to-end latency and cost figures report a single value per configuration, with no error bars, confidence intervals, or repeated runs. Because the workloads are synthetic Poisson arrivals with random timing, the p99 and average latency estimates are subject to sampling variability, and the claimed 'up to 1.44x' improvements could be within noise for some configurations. Please report the number of trials, the variance, and statistical significance (or at least inter-quartile ranges) for the headline numbers, especially for the burst and multi-node scenarios.","section":"Section 6.2 and Figures 10-12"},{"comment":"The 'theoretical optimum' in Algorithm 1 uses the same profiled execution times that DDiT itself uses to set its optimal DoP values (line 18: 'we pre-profile across various scenarios and leverage these data to estimate the execution time'). Consequently, the claim in Section 6.2 that DDiT reaches 1.39x of the theoretical optimum while the best baseline reaches 2.08x is not an independent lower bound; it reflects the fitted profiling table. This does not invalidate the latency comparisons against the baselines, but the cost-optimality comparison should be framed as an internal consistency check, not as a validation against an a-priori optimum.","section":"Section 4.2.2, Algorithm 1, and Section 6.2"}],"minor_comments":[{"comment":"There is a typo: 'monetray cost' should be 'monetary cost'.","section":"Section 6.1, Metrics"},{"comment":"The sentence 'The remaining GPUs continue to execute until the the request is completed' contains a duplicated 'the'.","section":"Section 4.3, lifecycle description"},{"comment":"The VideoSys baseline is cited as reference [43] in the text but as reference [54] in the implementation section; please reconcile the citation numbering.","section":"References [43] and [54]"},{"comment":"The text says 'batching can not improve the throughput' but Figure 3's caption is 'The impact of changing the batch size on the performance of T2V serving.' Please align the caption and the conclusion, and clarify which quantity (throughput or per-request latency) is being discussed.","section":"Section 3.1 and Figure 3"},{"comment":"The definition of starvation time depends on 'r_opt_step_time', but the text does not explain how the optimal per-step time is computed from the profiling data when multiple DoP values have similar per-step times. Adding one sentence of clarification would help.","section":"Section 4.2.3, Equation (5)"},{"comment":"The set of baselines consists only of VideoSys variants. The related work cites other T2V serving systems such as DistriFusion [20], xDiT [8], and SwiftDiffusion [22]; adding at least one non-VideoSys baseline, or explaining why they are not comparable, would strengthen the 'state-of-the-art baselines' claim.","section":"Section 6.1, Baselines"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the core idea of phase-decoupled, step-level elastic parallelism is timely. However, the internal contradiction around batching is the kind of issue that a careful reviewer will catch immediately; if the authors can add a batching-aware baseline or substantially revise Insight 1, the paper could become a solid systems contribution. The multi-node emulation also needs much more detail before the scalability claims can be trusted. I would not reject the paper, but I would require a full revision addressing the batching issue, the emulation methodology, and the statistical grounding of the headline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real systems contribution with a genuine new mechanism—changing the degree of sequence parallelism at single denoising-step granularity—and a broad, mostly careful evaluation. The headline 1.44x/1.43x latency numbers are plausible for the tested setup. But the paper's founding assumption that batching cannot help T2V is contradicted by its own Figure 17, and the multi-node results are emulated on one server. It deserves serious peer review, not a desk reject, but it needs major revision.\n\nWhat's new: DDiT decouples DiT and VAE, profiles an optimal degree of parallelism per resolution, and promotes/demotes GPUs at step granularity using a starvation-time priority. That is a real step beyond static DoP and request-level scheduling. The evaluation is extensive: four baselines, ten workload mixtures, several arrival rates, ablations for decoupling and promotion, and a theoretical-optimum comparison. The implementation is non-trivial (engine controller, lazy communicator construction, buddy allocator), and the cost model is clearly tied to GPU occupancy time. Credit where due: this is a working system, not just a simulation.\n\nSoft spots, in order of seriousness. First, Insight 1 in Section 3.1 is not supported and is internally contradicted. The text says batching cannot improve throughput, but Figure 17 shows DiT at 144p with parallel size 1: batch size 1 takes 3.44s for one request, batch size 2 takes 3.67s for two. That is roughly 1.9x throughput. The text even admits throughput scales linearly at smaller resolutions before concluding the opposite. This matters because the FCFS, no-batching scheduler and all four baselines exclude a batching-aware competitor. I don't think batching would erase DDiT's gains—step-level promotion is orthogonal to batching—but the comparison is incomplete and the justification is wrong. Second, the multi-node experiment is an emulation on a single 8-GPU server, not an actual 64-GPU cluster. The authors don't hide it, but the headline 30.4% reduction is only as strong as that emulation. Third, no code or data artifacts, no error bars, and only synthetic Poisson workloads. These are all addressable.\n\nThe theoretical-optimum comparison is fine as an idealized bound; it uses DDiT's own profiled timings, but it is clearly labeled as a model. The fitted parameters (optimal DoP per resolution, VAE DoP) do not make the central result circular, since the main comparisons are against fixed-deployment baselines.\n\nWho this is for: anyone building or evaluating T2V serving systems, and anyone working on elastic resource allocation for diffusion inference. It is a useful paper even if the batching question changes some of the numbers. Recommendation: send to peer review; reviewers should press for a batching-aware baseline, real multi-node validation, and released artifacts.","headline":"A credible T2V serving system with a genuinely new step-level resource elasticity mechanism, but the no-batching premise is contradicted by the paper's own Figure 17 and the multi-node results are emulated, so it needs major revision before acceptance.","tokens_in":22583,"tokens_out":1967,"would_cite":false,"duration_ms":19217,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DDiT claims that serving text-to-video cuts p99 latency by up to 30.4% by decoupling the DiT and VAE phases and reallocating GPU parallelism at single-step granularity.","keywords":["diffusion transformer serving","text-to-video generation","dynamic resource allocation","degree of parallelism","sequence parallelism","step-granularity scheduling","DiT-VAE decoupling","starvation time"],"falsifier":"Run a continuous-batching scheduler that co-executes several 144p or 240p requests on the same GPUs, and compare sustained throughput and p99 latency against DDiT's sequential FCFS policy on the same hardware and workload; if batching lifts throughput without worsening tail latency, the sequential-only assumption that motivates the whole design fails. A second check: use a VAE implementation that is actually accelerated by parallelism or batching; if VAE time stops being flat in DoP, the decoupling asymmetry DDiT exploits narrows.","tokens_in":21416,"feed_emoji":"🎬","tokens_out":14638,"duration_ms":128968,"temperature":0.7,"pith_summary":"DDiT argues that the standard way of serving text-to-video (T2V) models wastes GPU time, because the two heavy phases — the denoising Diffusion Transformer (DiT) and the video-decoding VAE — respond to parallelism in opposite ways, yet serving systems give both the same fixed degree of parallelism for the whole life of a request. The paper claims that decoupling the phases (pre-loading weights on all GPUs and building parallel communication groups only on demand), together with a greedy scheduler that changes parallelism at single-step granularity, lets released GPUs be handed to the next 'hungry' request mid-generation. On the OpenSora stack with a T5 text encoder, this reduces p99 latency by up to 30.4% and average latency by 30% in multi-node runs, and keeps cumulative GPU-occupancy cost at 1.39x of a theoretical optimum that knows the full request mix, where the best baseline sits at 2.08x. The work matters because T2V serving today is built on monolithic, static deployments, and the paper's measurements locate the waste and supply a working mechanism to recover it.","feed_headline":"Reallocating GPUs mid-generation cuts video-AI latency by 30 percent","feed_subtitle":"A step-level scheduler hands freed GPUs to waiting denoising jobs instead of letting them sit idle.","key_machinery":"The load-bearing pieces are (1) the engine controller, which separates model-weight loading from communication-group construction so that a running DiT instance can adopt new GPUs mid-execution by broadcasting latent tensors over NCCL in under a millisecond; (2) the offline-profiled B value, the optimal degree of parallelism for a resolution, derived from the per-step DiT time change rate $z$ between adjacent parallel sizes; and (3) the greedy scheduler, which starts a request on whatever free GPUs exist when its B target is unavailable and promotes 'hungry' requests by starvation time — the cumulative extra step time a request has suffered from running below its optimal parallelism. Together these make step-wise elastic parallelism practical: each reconfiguration costs less than a millisecond against DiT steps that take seconds, which is what turns the empirical scaling curves into a working scheduler.","core_discovery":"The central claim is that GPU resources for DiT-based T2V serving should be treated as a pool that can be attached to and detached from a running request at the granularity of a single denoising step, because the pipeline phases scale differently: DiT per-step time initially halves when the degree of parallelism doubles, with a resolution-specific sweet spot (the profiled 'B' values of 1, 2, and 4 GPUs for 144p, 240p, and 360p), while VAE time is essentially flat in parallelism. From this the paper concludes that homogeneous and statically partitioned deployments waste both the GPUs left idle between requests and the GPUs VAE cannot exploit, and that the fix is an elastic engine controller that scales parallelism up and down on demand around a First-Come-First-Serve, no-batching schedule. The claimed payoff is an up-to-1.44x improvement in p99 latency over the state-of-the-art baseline (at least 30.4% p99 and 30% average-latency reductions across four baselines in multi-node experiments), delivered while staying within 1.39x of the offline optimum's cumulative GPU occupancy time.","pith_inferences":["The no-batching assumption is the point most worth stress-testing: it rests on saturation measurements of one model family (OpenSora) at three resolutions on one 8-GPU node, and if continuous batching ever proves effective for T2V on other models or at higher resolutions, DDiT's gains would need to be re-measured against a batching-aware baseline.","The same decouple-and-profile recipe transfers to other generative pipelines with phase-asymmetric scaling, such as text-to-image DiT serving with a decoder that does not parallelize, or audio and 3D diffusion pipelines.","Starvation time can be generalized from 'time spent below optimal parallelism' to 'time spent below the parallelism a per-request deadline requires,' which would give commercial T2V APIs a deadline-aware admission and scheduling policy that the paper does not explore.","A sliding-window implementation of the paper's theoretical-optimum algorithm, recomputing allocations from recent arrival statistics, is a natural online extension; the paper leaves prediction-based allocation out because the methods it cites are sensitive to load fluctuations."],"forward_implications":["When the request mix shifts, static or cluster-isolated deployments strand GPUs that could be working on the next high-resolution request; DDiT's step-level promotion recaptures those GPUs, which is the mechanism behind the roughly 30% latency and cost reductions at high arrival rates and in bursts.","The profiled B values turn resolution-specific optimal parallelism into a deployable policy: one offline profiling run per resolution, then online scheduling consults the stored table.","Even though DDiT's greedy scheduler does not know the incoming request distribution, its cumulative GPU-occupancy cost reaches 1.39x of the theoretical-optimum value while the best distribution-aware baseline reaches 2.08x, indicating most of the achievable gain is captured without prediction.","Since scale-up and tensor transfer take under a millisecond while a DiT step takes seconds, dynamic parallelism changes are effectively free at the tested scales, which is why per-step reconfiguration is a practical rather than academic mechanism."],"supporting_citations":[{"why":"VideoSys, the open-source T2V serving system whose engines DDiT builds on and that serves as the primary baseline for the end-to-end latency and cost comparisons.","marker":"[43]"},{"why":"OpenSora, the model family (STDiT DiT, OpenSora VAE, T5 encoder) whose profiling measurements motivate the decoupling and produce the per-resolution optimal-parallelism values.","marker":"[25]"},{"why":"Ring-Attention sequence parallelism, the technique behind the elastic per-step parallel deployment that DDiT dynamically reconfigures.","marker":"[26]"},{"why":"Cited as Amdahl's Law, the scaling argument used to justify the existence of a per-resolution optimal degree of parallelism and the starvation-time accounting.","marker":"[10]"},{"why":"The Diffusion Transformer (DiT) architecture that defines the denoising phase whose per-step execution time is the object of all scheduling decisions.","marker":"[33]"}],"fun_headline_variants":["Swap GPUs per denoising step to cut T2V latency by 30%","Dynamic GPU pool for DiT serving: 1.44x p99 latency win","Single-step resource scheduling for diffusion transformer serving","Elastic DiT serving: match parallelism per resolution, cut latency","DDiT: attach GPUs per step to slash video AI latency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design assumes that batching never helps text-to-video serving — that a single request already saturates the GPUs, so processing requests one at a time without co-batching is the optimal policy — a conclusion drawn from measurements of one model (OpenSora) on one 8-GPU testbed at three resolutions.","fun_headline_variants_meta":{"raw":{"variants":["Swap GPUs per denoising step to cut T2V latency by 30%","Dynamic GPU pool for DiT serving: 1.44x p99 latency win","Single-step resource scheduling for diffusion transformer serving","Elastic DiT serving: match parallelism per resolution, cut latency","DDiT: attach GPUs per step to slash video AI latency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000812,"raw_usage":{"total_tokens":3611,"prompt_tokens":1048,"completion_tokens":2563,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":2467}},"tokens_in":664,"tokens_out":2563,"duration_ms":17744,"temperature":1.0,"reasoning_tokens":2467,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:59:47.962376+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a continuous-batching scheduler that co-executes several 144p or 240p requests on the same GPUs, and compare sustained throughput and p99 latency against DDiT's sequential FCFS policy on the same hardware and workload; if batching lifts throughput without worsening tail latency, the sequential-only assumption that motivates the whole design fails. A second check: use a VAE implementation that is actually accelerated by parallelism or batching; if VAE time stops being flat in DoP, the decoupling asymmetry DDiT exploits narrows.","supporting_citations":[{"cited_title":"Videosys: An easy and efficient system for video generation, 2024","cited_arxiv_id":null,"evidence_quote":"VideoSys, the open-source T2V serving system whose engines DDiT builds on and that serves as the primary baseline for the end-to-end latency and cost comparisons."},{"cited_title":"Reevaluating amdahl’s law.Communications of the ACM, 31(5):532–533, 1988","cited_arxiv_id":null,"evidence_quote":"Cited as Amdahl's Law, the scaling argument used to justify the existence of a per-resolution optimal degree of parallelism and the starvation-time accounting."}],"review_version":2}