Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

DDiT: Dynamic Resource Allocation for Diffusion Transformer Model Serving

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DDiT claims that serving text-to-video cuts p99 latency by up to 30.4% by decoupling the DiT and VAE phases and reallocating GPU parallelism at single-step granularity.

desk verdict A credible T2V serving system with a genuinely new step-level resource elasticity mechanism, but the no-batching premise is contradicted by the paper's own Figure 17 and the multi-node results are emulated, so it needs major revision before acceptance. read the letter →

arxiv 2506.13497 v1 pith:BZ4L5Q4C submitted 2025-06-16 cs.DC

Heyang Huang , Cunchen Hu , Jiaqi Zhu , Ziyuan Gao , Liangliang Xu , Yizhou Shan , Yungang Bao , Sun Ninghui
show 2 more authors
Tianwei Zhang Sa Wang
This is my paper · ORCID
classification cs.DC
keywords diffusiontransformerservingtext-to-videogenerationdynamicresourceallocationdegreeofparallelismsequencestep-granularityschedulingDiT-VAEdecouplingstarvationtime
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DDiT argues that the standard way of serving text-to-video (T2V) models wastes GPU time, because the two heavy phases — the denoising Diffusion Transformer (DiT) and the video-decoding VAE — respond to parallelism in opposite ways, yet serving systems give both the same fixed degree of parallelism for the whole life of a request. The paper claims that decoupling the phases (pre-loading weights on all GPUs and building parallel communication groups only on demand), together with a greedy scheduler that changes parallelism at single-step granularity, lets released GPUs be handed to the next 'hungry' request mid-generation. On the OpenSora stack with a T5 text encoder, this reduces p99 latency by up to 30.4% and average latency by 30% in multi-node runs, and keeps cumulative GPU-occupancy cost at 1.39x of a theoretical optimum that knows the full request mix, where the best baseline sits at 2.08x. The work matters because T2V serving today is built on monolithic, static deployments, and the paper's measurements locate the waste and supply a working mechanism to recover it.

What carries the argument

The load-bearing pieces are (1) the engine controller, which separates model-weight loading from communication-group construction so that a running DiT instance can adopt new GPUs mid-execution by broadcasting latent tensors over NCCL in under a millisecond; (2) the offline-profiled B value, the optimal degree of parallelism for a resolution, derived from the per-step DiT time change rate $z$ between adjacent parallel sizes; and (3) the greedy scheduler, which starts a request on whatever free GPUs exist when its B target is unavailable and promotes 'hungry' requests by starvation time — the cumulative extra step time a request has suffered from running below its optimal parallelism. Together these make step-wise elastic parallelism practical: each reconfiguration costs less than a millisecond against DiT steps that take seconds, which is what turns the empirical scaling curves into a working scheduler.

What would settle it

Run a continuous-batching scheduler that co-executes several 144p or 240p requests on the same GPUs, and compare sustained throughput and p99 latency against DDiT's sequential FCFS policy on the same hardware and workload; if batching lifts throughput without worsening tail latency, the sequential-only assumption that motivates the whole design fails. A second check: use a VAE implementation that is actually accelerated by parallelism or batching; if VAE time stops being flat in DoP, the decoupling asymmetry DDiT exploits narrows.

Watch

Extended reading notes

Core claim

The central claim is that GPU resources for DiT-based T2V serving should be treated as a pool that can be attached to and detached from a running request at the granularity of a single denoising step, because the pipeline phases scale differently: DiT per-step time initially halves when the degree of parallelism doubles, with a resolution-specific sweet spot (the profiled 'B' values of 1, 2, and 4 GPUs for 144p, 240p, and 360p), while VAE time is essentially flat in parallelism. From this the paper concludes that homogeneous and statically partitioned deployments waste both the GPUs left idle between requests and the GPUs VAE cannot exploit, and that the fix is an elastic engine controller that scales parallelism up and down on demand around a First-Come-First-Serve, no-batching schedule. The claimed payoff is an up-to-1.44x improvement in p99 latency over the state-of-the-art baseline (at least 30.4% p99 and 30% average-latency reductions across four baselines in multi-node experiments), delivered while staying within 1.39x of the offline optimum's cumulative GPU occupancy time.

Load-bearing premise

The design assumes that batching never helps text-to-video serving — that a single request already saturates the GPUs, so processing requests one at a time without co-batching is the optimal policy — a conclusion drawn from measurements of one model (OpenSora) on one 8-GPU testbed at three resolutions.

Editorial extensions

If this is right

  • When the request mix shifts, static or cluster-isolated deployments strand GPUs that could be working on the next high-resolution request; DDiT's step-level promotion recaptures those GPUs, which is the mechanism behind the roughly 30% latency and cost reductions at high arrival rates and in bursts.
  • The profiled B values turn resolution-specific optimal parallelism into a deployable policy: one offline profiling run per resolution, then online scheduling consults the stored table.
  • Even though DDiT's greedy scheduler does not know the incoming request distribution, its cumulative GPU-occupancy cost reaches 1.39x of the theoretical-optimum value while the best distribution-aware baseline reaches 2.08x, indicating most of the achievable gain is captured without prediction.
  • Since scale-up and tensor transfer take under a millisecond while a DiT step takes seconds, dynamic parallelism changes are effectively free at the tested scales, which is why per-step reconfiguration is a practical rather than academic mechanism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The no-batching assumption is the point most worth stress-testing: it rests on saturation measurements of one model family (OpenSora) at three resolutions on one 8-GPU node, and if continuous batching ever proves effective for T2V on other models or at higher resolutions, DDiT's gains would need to be re-measured against a batching-aware baseline.
  • The same decouple-and-profile recipe transfers to other generative pipelines with phase-asymmetric scaling, such as text-to-image DiT serving with a decoder that does not parallelize, or audio and 3D diffusion pipelines.
  • Starvation time can be generalized from 'time spent below optimal parallelism' to 'time spent below the parallelism a per-request deadline requires,' which would give commercial T2V APIs a deadline-aware admission and scheduling policy that the paper does not explore.
  • A sliding-window implementation of the paper's theoretical-optimum algorithm, recomputing allocations from recent arrival statistics, is a natural online extension; the paper leaves prediction-based allocation out because the methods it cites are sensitive to load fluctuations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents DDiT, a serving system for text-to-video (T2V) diffusion transformers. DDiT decouples the DiT and VAE phases into independent elastic units, determines an optimal degree of sequence parallelism (DoP) per resolution from offline profiling, and schedules DiT execution at single-step granularity using a greedy algorithm that promotes the DoP of 'hungry' requests based on a starvation-time priority. The evaluation, using OpenSora models and VideoSys-derived baselines, reports that DDiT outperforms the baselines by up to 1.44x in p99 latency and 1.43x in average latency (single-node and multi-node, the latter being an emulation), with a breakdown of the DiT-VAE decoupling and DoP promotion mechanisms, plus a comparison against a theoretical optimal scheduling algorithm.

Significance. If the empirical results hold, the paper makes a useful systems contribution to T2V serving. The strengths are: (1) the observation that DiT and VAE have distinct parallelism behaviors and that a single static DoP is suboptimal across mixed resolutions and arrival rates; (2) the design of step-level DoP promotion with measured overhead below 1 ms (Figure 15); and (3) a broad ablation split (DiT-VAE decoupling, DoP promotion) that isolates the benefit of each mechanism. The paper also ships reproducible pseudocode for its scheduling algorithms. However, the headline claims rest on a load-bearing premise about batching that is internally contradicted by the paper's own experimental data, and the multi-node evaluation is an emulation on a single server. These issues materially affect whether the claimed gains would persist against a more complete set of serving baselines.

major comments (4)
  1. [Section 3.1 and Figure 17] Insight 1, which motivates the FCFS no-batching scheduler (Section 4.2.3) and the absence of batching-aware baselines in Figures 10-12, is not supported by the paper's own measurements. The text states that 'the throughput scales linearly with the batch size at smaller resolutions' and then concludes 'Batching can not improve the throughput of the T2V system.' Figure 17 directly contradicts this: at 144p with parallel size 1, DiT latency is 3.44 s for batch 1 and 3.67 s for batch 2, i.e., per-request throughput rises from about 0.29 req/s to 0.55 req/s; at 240p the throughput also increases with batch size. If batching improves throughput for low and medium resolutions, then sequential FCFS is not obviously optimal, and the comparison set omits a plausible batching-aware competitor. Please either revise Insight 1 to reflect the measured superlinear throughput at small batch sizes, or add a batching-capable baseline (e.g., continuous batching) to the end-to-end evaluation to show that the headline latency gains are not an artifact of excluding such a system.
  2. [Section 6.1 'Testbed' and Section 6.2 'Multi-Node Performance'] The multi-node experiments, which are the basis for the claims of at least 30.4% p99 and 30% average latency reduction, are described as an emulation of an 8-node, 64-GPU cluster on a single server with eight H800 GPUs interconnected via NVLink. The manuscript does not explain how the 200 Gbps RDMA inter-node network, cross-node communication overhead, and multi-node topology are emulated. Without this detail, the multi-node results in Figures 11 and 12 cannot be interpreted as evidence of scalability; they may only reflect single-node behavior with artificial partitioning. Please describe the emulation methodology precisely or report results on a real multi-node cluster.
  3. [Section 6.2 and Figures 10-12] All end-to-end latency and cost figures report a single value per configuration, with no error bars, confidence intervals, or repeated runs. Because the workloads are synthetic Poisson arrivals with random timing, the p99 and average latency estimates are subject to sampling variability, and the claimed 'up to 1.44x' improvements could be within noise for some configurations. Please report the number of trials, the variance, and statistical significance (or at least inter-quartile ranges) for the headline numbers, especially for the burst and multi-node scenarios.
  4. [Section 4.2.2, Algorithm 1, and Section 6.2] The 'theoretical optimum' in Algorithm 1 uses the same profiled execution times that DDiT itself uses to set its optimal DoP values (line 18: 'we pre-profile across various scenarios and leverage these data to estimate the execution time'). Consequently, the claim in Section 6.2 that DDiT reaches 1.39x of the theoretical optimum while the best baseline reaches 2.08x is not an independent lower bound; it reflects the fitted profiling table. This does not invalidate the latency comparisons against the baselines, but the cost-optimality comparison should be framed as an internal consistency check, not as a validation against an a-priori optimum.
minor comments (6)
  1. [Section 6.1, Metrics] There is a typo: 'monetray cost' should be 'monetary cost'.
  2. [Section 4.3, lifecycle description] The sentence 'The remaining GPUs continue to execute until the the request is completed' contains a duplicated 'the'.
  3. [References [43] and [54]] The VideoSys baseline is cited as reference [43] in the text but as reference [54] in the implementation section; please reconcile the citation numbering.
  4. [Section 3.1 and Figure 3] The text says 'batching can not improve the throughput' but Figure 3's caption is 'The impact of changing the batch size on the performance of T2V serving.' Please align the caption and the conclusion, and clarify which quantity (throughput or per-request latency) is being discussed.
  5. [Section 4.2.3, Equation (5)] The definition of starvation time depends on 'r_opt_step_time', but the text does not explain how the optimal per-step time is computed from the profiling data when multiple DoP values have similar per-step times. Adding one sentence of clarification would help.
  6. [Section 6.1, Baselines] The set of baselines consists only of VideoSys variants. The related work cites other T2V serving systems such as DistriFusion [20], xDiT [8], and SwiftDiffusion [22]; adding at least one non-VideoSys baseline, or explaining why they are not comparable, would strengthen the 'state-of-the-art baselines' claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the fitted DoP values and the profiled theoretical optimum are system parameters or idealized bounds, while the headline latency claims are measured comparisons against fixed-deployment baselines.

full rationale

The paper's central derivation chain is empirical: it measures DiT and VAE latency under different degrees of parallelism (Figure 5), defines a heuristic optimal DoP B from the largest per-doubling speedup (Equation 4), and then evaluates DDiT by direct measurement against fixed-deployment baselines (Figures 10-12). The B values and the VAE DoP of one are fitted from profiling data, but they are system parameters, not predicted outcomes: the headline latency reductions are measured end-to-end comparisons, so they do not reduce by construction to the fitted values. Algorithm 1's 'theoretical optimum' is computed from the same profiled per-request execution times and therefore is an idealized lower-bound benchmark rather than an independent prediction; this weakens its value as a validation tool, but it is not the paper's central empirical claim, and comparing DDiT to a lower bound computed from measured service times remains a meaningful sanity check. Self-citations [15,16] appear only as related-work motivation for disaggregation and do not carry any load-bearing uniqueness or correctness argument. The internal tension in Insight 1 about batching (Figure 17 shows per-request throughput rising with batch size at low resolutions) is a correctness or experimental-design concern, not circularity: it does not make any equation equal to its own input. No step in the derivation is equivalent to its inputs by definition, so no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central claim rests on profiled B values, the sequential-processing observation, VAE non-scalability, the starvation-time heuristic, and queueing approximations. The first three are hardware- and model-specific empirical inputs; the fourth is an ad hoc priority rule; the fifth is standard approximation used only for the ideal benchmark. No new physical or abstract entities are introduced.

free parameters (2)
  • Optimal DoP B per resolution = 144p: 1, 240p: 2, 360p: 4
    Chosen in Section 4.2.3 from Figure 8 as the parallel size with the largest per-step latency reduction z; this hardware-specific fit drives the greedy scheduler and is not derived from first principles.
  • VAE DoP = 1
    Set to one for all requests based on the observation in Section 3.2 that VAE latency does not decrease with parallelism; this is a design choice fitted to the profiled VAE behavior.
assumptions (6)
  • domain assumption DiT step execution time is deterministic for a fixed (resolution, DoP) and accurately captured by offline profiling.
    Used in Algorithm 1 line 18 and in the B-value determination (Section 4.2.2, 4.2.3); stale profiles or different hardware would make the chosen 'optimal' DoP suboptimal.
  • domain assumption Batching does not improve T2V throughput, so sequential FCFS execution is the optimal scheduling policy.
    Insight 1 in Section 3.1, based on batch-size 1-8 experiments on OpenSora; this excludes batching-aware baselines and shapes the greedy scheduling design.
  • domain assumption The VAE stage cannot be accelerated by parallelization.
    Section 3.2 states VAE latency is stable across DoP and DistVAE does not help; this justifies VAE DoP=1 and the decoupling benefit.
  • ad hoc to paper The starvation-time metric (Equation 5) is a valid priority signal for maximizing GPU utilization.
    The scheduler ranks hungry requests by this heuristic; no proof of optimality is given, and the system's performance depends on this ranking being effective.
  • standard math Standard queueing approximations (M/D/1, M/M/c, Stirling's formula) are valid for the theoretical-optimum model.
    Appendix A uses these approximations to compute mean occupancy times in the DP; they introduce modeling error, so the 'theoretical optimum' is approximate.
  • domain assumption Each GPU can hold a full replica of the model weights.
    DDiT loads full weights on every GPU (Section 4.1); for larger models or lower-memory GPUs, tensor or pipeline parallelism would be needed, changing the design.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DDiT: Dynamic Resource Allocation for Diffusion Transformer Model Serving." pith.science (2026). https://pith.science/paper/BZ4L5Q4C

@misc{pith2026250613497,
  author       = {Pith},
  title        = {Pith review of: DDiT: Dynamic Resource Allocation for Diffusion Transformer Model Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BZ4L5Q4C}},
  note         = {Machine review of arXiv:2506.13497}
}
read the original abstract

The Text-to-Video (T2V) model aims to generate dynamic and expressive videos from textual prompts. The generation pipeline typically involves multiple modules, such as language encoder, Diffusion Transformer (DiT), and Variational Autoencoders (VAE). Existing serving systems often rely on monolithic model deployment, while overlooking the distinct characteristics of each module, leading to inefficient GPU utilization. In addition, DiT exhibits varying performance gains across different resolutions and degrees of parallelism, and significant optimization potential remains unexplored. To address these problems, we present DDiT, a flexible system that integrates both inter-phase and intra-phase optimizations. DDiT focuses on two key metrics: optimal degree of parallelism, which prevents excessive parallelism for specific resolutions, and starvation time, which quantifies the sacrifice of each request. To this end, DDiT introduces a decoupled control mechanism to minimize the computational inefficiency caused by imbalances in the degree of parallelism between the DiT and VAE phases. It also designs a greedy resource allocation algorithm with a novel scheduling mechanism that operates at the single-step granularity, enabling dynamic and timely resource scaling. Our evaluation on the T5 encoder, OpenSora SDDiT, and OpenSora VAE models across diverse datasets reveals that DDiT significantly outperforms state-of-the-art baselines by up to 1.44x in p99 latency and 1.43x in average latency.

Figures

Figures reproduced from arXiv: 2506.13497 by the authors.

Figure 1
Figure 1. The deploy architecture of a T2V serving system. the input token length 𝐿, while DiT needs to process a large number of tokens in each denoising step [4, 51]. It is important but challenging to develop an efficient T2V serving system with low latency and high resource utiliza￾tion [20, 22]. A number of techniques have been proposed to accelerate the video generation process, including distilla￾tion [51], post-traini… view at source ↗
Figure 2
Figure 2. The architecture of T2V systems. users [18, 31, 37]. However, higher resolutions lead to a qua￾dratic increase in computational complexity, significantly affecting the generation latency [51]. 2.2 Text-to-Video Generation Text-to-Video (T2V) [49, 55, 56] is a multimodal generation task that produces videos from text descriptions, often re￾quiring integrating multiple collaborative modules. We use one of the most pop… view at source ↗
Figure 5
Figure 5. Latency of DiT and VAE under various DoPs. However, VAE maintains stable execution time regardless of GPU parallelization. As the DoP increases, the latency of VAE accounts for a larger proportion of the total serv￾ing time, leading to greater resource inefficiency with more GPUs. For example, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figures from the paper (8 more)
Figure 6
Figure 6. Figure 6: The idle GPU under the fixed deployment. across varying DoP configurations, with slight degradation at higher DoP; (2) for higher-resolution requests, the perfor￾mance improves significantly as DoP increases. For instance, a request with 144p resolution performs better…
Figure 8
Figure 8. Figure 8: Per-step DiT time change rate between adjacent parallel sizes. the different resource requirements of components like the DiT and VAE modules. DDiT decouples DiT and VAE into two independent components, enabling them to execute se￾quentially within the same instance at…
Figure 10
Figure 10. Figure 10: Single-node end-to-end performance (Lower latency indicates better performance). Part is an abbreviation for Partition, and Isol is isolation for short. Metrics. We focus on the average latency, the p99 latency, and the monetary cost of a T2V inference system. For eac…
Figure 11
Figure 11. Figure 11: Multi-node end-to-end performance. 1: 1: 1 2: 2: 6 2: 6: 2 6: 2: 2 2: 4: 4 4: 2: 4 4: 4: 2 1: 3: 6 6: 1: 3 3: 6: 1 Low: Medium: High 0.2 0.4 0.6 0.8 1.0 Normalized Cost Theoretical Optimum DDiT Static DoP Static Part. & Cluster Isol. Dynamic Part. & Cluster Isol. Dyna…
Figure 12
Figure 12. Figure 12: Multi-node monetary cost (Lower Cost indicates a better solution). DP, cross-cluster resource sharing completely compensates for the shortage of model instances. At lower arrival rates (0.25 and 0.5), DDiT achieves a 4.9% reduction in p99 latency. This improvement inc…
Figure 15
Figure 15. Figure 15: , it introduces less than 1 millisecond of delay. Com￾pared to the several seconds to tens of seconds required for DiT execution, these overheads are negligible. 7 Related Work Efficient Transformer Serving. Existing transformer serv￾ing systems utilize efficient GPU …
Figure 16
Figure 16. Figure 16: Single-node end-to-end performance II. Time (s) 3.01 5.50 10.43 20.08 8.28 15.87 30.97 61.31 19.17 37.52 74.51 148.40 39.64 78.69 156.57 00M DiT Latency (Parallel Size = 1) DiT_144p DiT_240p DiT_360p DiT_480p 0.16 0.30 0.59 1.17 0.38 0.75 1.48 2.93 0.87 1.68 3.31 6.58…
Figure 17
Figure 17. Figure 17 [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OpScale: Operator-level Provisioning and Autoscaling for LLM Serving

    cs.DC 2026-08 conditional novelty 7.0 of 10

    Operator-level autoscaling meets LLM latency targets with 20-36% fewer GPUs and 14-28% less power than model-level replica scaling.

  2. TetriServe: Efficiently Serving Mixed DiT Workloads

    cs.LG 2025-10 conditional novelty 6.0 of 10

    TetriServe's step-level, deadline-aware sequence parallelism improves SLO attainment for mixed-resolution diffusion transformer serving by up to 32% over fixed-SP systems.

Reference graph

Works this paper leans on

57 extracted references · 24 canonical work pages · cited by 2 Pith papers

  1. [1]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. InSC22: International Conference for High Performance Computing, Networking, Storage and Analysis, 2022

    work page 2022

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendele- vitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  3. [3]

    Striped atten- tion: Faster ring attention for causal transformers.arXiv preprint arXiv:2311.09431, 2023

    William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. Striped atten- tion: Faster ring attention for causal transformers.arXiv preprint arXiv:2311.09431, 2023

    arXiv 2023

  4. [4]

    Pixart- 𝑠𝑖𝑔𝑚𝑎: Weak-to-strong training of diffusion transformer for 4k text- to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- 𝑠𝑖𝑔𝑚𝑎: Weak-to-strong training of diffusion transformer for 4k text- to-image generation. InEuropean Conference on Computer Vision, pages 74–91. Springer, 2025

    work page 2025

  5. [5]

    arXiv preprint arXiv:2406.01125, 2024

    Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, and Tao Chen.𝑑𝑒𝑙𝑡𝑎 -dit: A training-free acceleration method tailored for diffusion transformers. arXiv preprint arXiv:2406.01125, 2024

    arXiv 2024

  6. [6]

    Flashattention: Fast and memory-efficient exact attention with io- awareness.Advances in Neural Information Processing Systems, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io- awareness.Advances in Neural Information Processing Systems, 2022

    work page 2022

  7. [7]

    Addison-Wesley, 2013

    Paul DuBois.MySQL. Addison-Wesley, 2013

    work page 2013

  8. [8]

    xdit: an inference engine for diffusion transformers (dits) with massive parallelism.arXiv preprint arXiv:2411.01738, 2024

    Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. xdit: an inference engine for diffusion transformers (dits) with massive parallelism.arXiv preprint arXiv:2411.01738, 2024

    arXiv 2024

Show all 57 references
  1. [9]

    Google.https://huggingface.co/google/t5-v1_1-xxl

  2. [10]

    Reevaluating amdahl’s law.Communications of the ACM, 31(5):532–533, 1988

    John L Gustafson. Reevaluating amdahl’s law.Communications of the ACM, 31(5):532–533, 1988

    1988

  3. [11]

    Denoising diffusion prob- abilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion prob- abilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  4. [12]

    Flashdecoding++: Faster large language model inference on gpus.arXiv preprint arXiv:2311.01282, 2023

    Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Hanyu Dong, and Yu Wang. Flashdecoding++: Faster large language model inference on gpus.arXiv preprint arXiv:2311.01282, 2023

    2023 arXiv

  5. [13]

    Hpcai Tech.https://huggingface.co/hpcai-tech/OpenSora-STDiT-v3

  6. [14]

    Hpcai Tech.https://huggingface.co/hpcai-tech/OpenSora-VAE-v1.2

  7. [15]

    Memserve: Context caching for disaggregated llm serving with elastic memory pool.arXiv preprint arXiv:2406.17565, 2024

    Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, et al. Memserve: Context caching for disaggregated llm serving with elastic memory pool.arXiv preprint arXiv:2406.17565, 2024

    2024 arXiv

  8. [16]

    Inference without interference: Disaggregate llm inference for mixed downstream workloads.arXiv preprint arXiv:2401.11181, 2024

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Inference without interference: Disaggregate llm inference for mixed downstream workloads.arXiv preprint arXiv:2401.11181, 2024

    2024 arXiv

  9. [17]

    Reducing activation recomputation in large transformer models

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catan- zaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5:341–353, 2023

    2023

  10. [18]

    KuaiShou.https://kling.kuaishou.com/

  11. [19]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, 2023

    2023

  12. [20]

    Distrifusion: Distributed parallel inference for high-resolution diffusion models

    Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2024

  13. [21]

    Sequence parallelism: Long sequence training from system per- spective.arXiv preprint arXiv:2105.13120, 2021

    Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system per- spective.arXiv preprint arXiv:2105.13120, 2021

    2021 arXiv

  14. [22]

    Swiftdiffusion: Efficient diffusion model serving with add-on modules.arXiv preprint arXiv:2407.02031, 2024

    Suyi Li, Lingyun Yang, Xiaoxiao Jiang, Hanfeng Lu, Zhipeng Di, Weiyi Lu, Jiawei Chen, Kan Liu, Yinghao Yu, Tao Lan, et al. Swiftdiffusion: Efficient diffusion model serving with add-on modules.arXiv preprint arXiv:2407.02031, 2024

    2024 arXiv

  15. [23]

    A survey of convolutional neural networks: analysis, applications, and prospects.IEEE transactions on neural networks and learning systems, 33(12):6999–7019, 2021

    Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou. A survey of convolutional neural networks: analysis, applications, and prospects.IEEE transactions on neural networks and learning systems, 33(12):6999–7019, 2021

    2021

  16. [24]

    Train big, then compress: Rethinking model size for efficient training and inference of transformers

    Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. Train big, then compress: Rethinking model size for efficient training and inference of transformers. InInterna- tional Conference on machine learning, 2020

    2020

  17. [25]

    Open- sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open- sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    2024 arXiv

  18. [26]

    Ring attention with blockwise transformers for near-infinite context.arXiv preprint arXiv:2310.01889, 2023

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context.arXiv preprint arXiv:2310.01889, 2023

    2023 arXiv

  19. [27]

    Deepcache: Accelerat- ing diffusion models for free

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerat- ing diffusion models for free. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15762–15772, 2024

    2024

  20. [28]

    Sentence-t5: Scalable sen- tence encoders from pre-trained text-to-text models.arXiv preprint arXiv:2108.08877, 2021

    Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sen- tence encoders from pre-trained text-to-text models.arXiv preprint arXiv:2108.08877, 2021

    2021 arXiv

  21. [29]

    The design process for google’s training chips: Tpuv2 and tpuv3.IEEE Micro, 41(2):56–63, 2021

    Thomas Norrie, Nishant Patil, Doe Hyun Yoon, George Kurian, Sheng Li, James Laudon, Cliff Young, Norman Jouppi, and David Patterson. The design process for google’s training chips: Tpuv2 and tpuv3.IEEE Micro, 41(2):56–63, 2021

    2021

  22. [30]

    NCCL.https://docs.nvidia.com/deeplearning/nccl/user- guide/docs/overview.html

    NVIDIA. NCCL.https://docs.nvidia.com/deeplearning/nccl/user- guide/docs/overview.html

  23. [31]

    Ocean.https://video-ocean.com/en

  24. [32]

    Splitwise: Efficient genera- tive llm inference using phase splitting.arXiv preprint arXiv:2311.18677, 2023

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient genera- tive llm inference using phase splitting.arXiv preprint arXiv:2311.18677, 2023

    2023 arXiv

  25. [33]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  26. [34]

    Variational autoencoder

    Lucas Pinheiro Cinelli, Matheus Araújo Marins, Eduardo Antúnio Barros da Silva, and Sérgio Lima Netto. Variational autoencoder. In Variational Methods for Machine Learning with Applications to Deep Networks, pages 111–149. Springer, 2021

    2021

  27. [35]

    Seagull: An infrastructure for load predic- tion and optimized resource allocation.arXiv preprint arXiv:2009.12922, 2020

    Olga Poppe, Tayo Amuneke, Dalitso Banda, Aritra De, Ari Green, Manon Knoertzer, Ehi Nosakhare, Karthik Rajendran, Deepak Shankar- gouda, Meina Wang, et al. Seagull: An infrastructure for load predic- tion and optimized resource allocation.arXiv preprint arXiv:2009.12922, 2020

    2009 arXiv

  28. [36]

    A remark on stirling’s formula.The American mathematical monthly, 62(1):26–29, 1955

    Herbert Robbins. A remark on stirling’s formula.The American mathematical monthly, 62(1):26–29, 1955

    1955

  29. [37]

    RunWay.https://runwayml.com/

  30. [38]

    Fora: Fast-forward caching in diffusion transformer 13 acceleration.arXiv preprint arXiv:2407.01425, 2024

    Pratheba Selvaraju, Tianyu Ding, Tianyi Chen, Ilya Zharkov, and Luming Liang. Fora: Fast-forward caching in diffusion transformer 13 acceleration.arXiv preprint arXiv:2407.01425, 2024

    2024 arXiv

  31. [39]

    Megatron-lm: Training multi- billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi- billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  32. [40]

    Fundamentals of queueing theory, volume 399

    John F Shortle, James M Thompson, Donald Gross, and Carl M Harris. Fundamentals of queueing theory, volume 399. John Wiley & Sons, 2018

    2018

  33. [41]

    Rpc: Remote procedure call protocol specification version 2

    Raj Srinivasan. Rpc: Remote procedure call protocol specification version 2. Technical report, 1995

    1995

  34. [42]

    One embedder, any task: Instruction-finetuned text embeddings

    Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction-finetuned text embeddings. arXiv preprint arXiv:2212.09741, 2022

    2022 arXiv

  35. [43]

    Videosys: An easy and efficient system for video generation, 2024

    VideoSys Team. Videosys: An easy and efficient system for video generation, 2024

    2024

  36. [44]

    New and old results for the m/d/c queue.AEU- International Journal of Electronics and Communications, 60(2):125–130, 2006

    Henk Tijms. New and old results for the m/d/c queue.AEU- International Journal of Electronics and Communications, 60(2):125–130, 2006

    2006

  37. [45]

    Pipefu- sion: Displaced patch pipeline parallelism for inference of diffusion transformer models.arXiv preprint arXiv:2405.14430, 2024

    Jiannan Wang, Jiarui Fang, Aoyu Li, and PengCheng Yang. Pipefu- sion: Displaced patch pipeline parallelism for inference of diffusion transformer models.arXiv preprint arXiv:2405.14430, 2024

    2024 arXiv

  38. [46]

    NVLink.https://en.wikipedia.org/wiki/NVLink

    Wikipedia. NVLink.https://en.wikipedia.org/wiki/NVLink

  39. [47]

    Cache me if you can: Accelerating diffusion models through block caching

    Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me if you can: Accelerating diffusion models through block caching. InProceedings of the IEEE/CVF Conference on Computer Vision...

    2024

  40. [48]

    Preact: Predictive resource allocation for bursty workloads in a co-located data center

    Dingyu Yang, Ziyang Xiao, Dongxiang Zhang, Shuhao Zhang, Jian Cao, and Gang Chen. Preact: Predictive resource allocation for bursty workloads in a co-located data center. InProceedings of the 53rd International Conference on Parallel Processing, pages 722–731, 2024

    2024

  41. [49]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  42. [50]

    Orca: A distributed serving system for {Transformer-Based} generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In16th USENIX Sympo- sium on Operating Systems Design and Implementation (OSDI 22), 2022

    2022

  43. [51]

    Ditfas- tattn: Attention compression for diffusion transformer models.arXiv preprint arXiv:2406.08552, 2024

    Zhihang Yuan, Hanling Zhang, Pu Lu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfas- tattn: Attention compression for diffusion transformer models.arXiv preprint arXiv:2406.08552, 2024

    2024 arXiv

  44. [52]

    Cross-attention makes in- ference cumbersome in text-to-image diffusion models.arXiv preprint arXiv:2404.02747, 2024

    Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, and Jürgen Schmidhuber. Cross-attention makes in- ference cumbersome in text-to-image diffusion models.arXiv preprint arXiv:2404.02747, 2024

    2024 arXiv

  45. [53]

    Dsp: Dynamic sequence parallelism for multi-dimensional transformers.arXiv preprint arXiv:2403.10266, 2024

    Xuanlei Zhao, Shenggan Cheng, Chang Chen, Zangwei Zheng, Ziming Liu, Zheming Yang, and Yang You. Dsp: Dynamic sequence parallelism for multi-dimensional transformers.arXiv preprint arXiv:2403.10266, 2024

    2024 arXiv

  46. [54]

    Real-time video generation with pyramid attention broadcast.arXiv preprint arXiv:2408.12588, 2024

    Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast.arXiv preprint arXiv:2408.12588, 2024

    2024 arXiv

  47. [55]

    Open-sora: Democratizing efficient video production for all, March 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024

    2024

  48. [56]

    Open- sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Sheng- gui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open- sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024

    2024 arXiv

  49. [57]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. arXiv preprint arXiv:2401.09670, 2024. 14 A Queue Model We clarify the queue mo...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.