Pith. sign in

REVIEW 2 major objections 4 minor 42 references

Zellige: Moldable Sequence Placement for Mixed Image-Video DiT Training

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that treating each sequence as a moldable task with overlapping rank sets removes the load-imbalance-versus-communication tradeoff and delivers 1.1–1.5x speedups in mixed image-video DiT training.

desk verdict Solid systems paper with a genuinely new per-sequence placement idea, sound formal tradeoffs, and plausible speedups—but the load-bearing additive time model is validated on a narrow set of plans. read the letter →

arxiv 2608.01150 v1 pith:L7DEPYNX submitted 2026-08-02 cs.DC

Guangyu Xiang , Xueze Kang , Minwei Zhao , Yuxin Wang , Shaohuai Shi , Lin Zhang , Xiaowen Chu This is my paper · ORCID
classification cs.DC
keywords DiffusionTransformersmixedimage-videotrainingsequenceplacementcontextparallelismloadbalancingcollectivecommunicationmoldabletasksdistributed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that mixed image-video DiT training suffers from a structural defect in existing placement schemes: letting each sequence choose only among disjoint rank groups forces a choice between some ranks sitting idle while others work, and short sequences paying for communication they do not need. It then presents Zellige, which lets each sequence pick both a parallelism configuration and an overlapping set of ranks, so whole short sequences can share a GPU with shards of long videos. The system's two-stage planner and coalesced attention engine aim to make this moldable placement practical, with a profiler accurate to 3.4% in step time and 1.5% in peak memory across 21 plans. In end-to-end runs, Zellige reports being 1.12–1.48x faster than the KnapFormer baseline on 16 A800 GPUs and 1.27–1.54x faster on 32 A6000 GPUs.

What carries the argument

The central object is moldable sequence placement: each training sequence is assigned a parallelism configuration $\theta=(q,h,k)$, splitting queries, attention heads, and key/value states, plus a participating rank set, and these sets may overlap across sequences. This is what lets a whole short image sequence occupy the same rank as a shard of a long video sequence, which disjoint-group schemes forbid. The paper's argument is carried by three mechanisms working together: a hardware profiler that reduces each bucket–configuration pair to an additive per-rank time price and an analytic memory cost; a two-stage planner that first solves an exact compact formulation of the anchor-placement problem with CP-SAT and then greedily packs whole filler sequences by a bottleneck score; and a coalesced attention engine that stably schedules and fuses whole sequences and distributed-attention shards on shared ranks. The role of the profiler is to make the planner's objective linear in the choice variables, and the role of the engine is to make the resulting co-located placements actually run at the profiled speed.

What would settle it

Measure a 22nd mixed batch—for instance four 10-second 720p videos plus 200 images on 16 A800 GPUs—and compare the measured step makespan with the profiler's additive prediction; an error well above the reported 3.4% MAPE, or a measured makespan no faster than KnapFormer's, would show the additive model does not transfer.

Watch

Extended reading notes

Core claim

Zellige's central claim is that the right unit of placement in mixed-image-video DiT training is the individual sequence, not the rank group. The paper formalizes existing DP/CP hybrids as disjoint-group placement and proves that any such scheme faces a tradeoff between inter-group load imbalance and intra-group communication redundancy: with disjoint groups, either a large sequence cannot spread beyond its group's ranks, or small sequences inside multi-rank groups are forced to communicate. Zellige instead treats each sequence as a moldable task whose parallelism configuration—query, head, and key/value splits—and participating rank set are chosen jointly, with rank sets allowed to overlap. A hardware profiler supplies per-configuration time and memory prices; a two-stage planner balances heavy anchor sequences with exact CP-SAT and packs light filler sequences whole into the remaining headroom; and a coalesced attention engine executes the resulting mixed placement. The reported end-to-end result is that this system is fastest in all twelve measured workloads, beating KnapFormer by 1.12–1.48x on 16 A800s and 1.27–1.54x on 32 A6000s.

Load-bearing premise

The load-bearing assumption is that a GPU's step time equals the sum of the individually profiled times of the sequences assigned to it, with no extra slowdown from co-locating whole sequences and split-video shards; the paper checks this on only 21 plan compositions.

Editorial extensions

If this is right

  • Disjoint-group placement, including pure DP and pure CP, cannot simultaneously achieve perfect load balance and avoid unnecessary communication; the paper's theorems quantify this as a worst-case makespan ratio of $N/g_{\max}$ and an unbounded redundancy in split count.
  • A batch can be planned fast enough to be useful per step: 33–119 ms solve times, with at most 0.32% modeled makespan loss versus a joint-placement reference.
  • Sequence co-location cuts attention communication volume: in the high-load V50 case, Zellige moves 25.4% of USP's attention data and 64.9% of KnapFormer's, while matching USP's max/mean rank-time ratio of 1.00x.
  • Zellige is fastest in all twelve end-to-end workloads tested, including both 10-second and 15-second video suites, and it is the only method that runs the 15-second workloads without OOM while AdaptiveLoad fails.
  • If the additive per-rank model holds, the same two-stage anchor/filler decomposition should remain near-optimal for any mixed batch that respects the profiler's bucket catalog.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The additive per-rank time model assumes no interference among whole sequences and distributed-attention shards co-located on a rank; if such interference grows with shard count or with smaller memory budgets, the planner's decisions and the measured speedups could erode on workloads outside the 21 validation plans.
  • The anchor/filler catalog cut is fixed per training setup; a dynamic per-batch cut might improve when the mix of image and video lengths shifts between runs or over the course of training.
  • The same moldable placement idea could apply to long-context LLM training jobs that mix very long and short sequences, where disjoint-group schemes face the same load-versus-communication tradeoff.
  • On clusters with different interconnect topology, the optimal split degree and the balance point between anchor splitting and filler packing would likely shift, since the profiler re-measures per setup but the two-stage decomposition is topology-agnostic.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces Zellige, a system for placing mixed image-video DiT training sequences onto GPU ranks. The key idea is to treat each sequence as a moldable task, jointly selecting its parallelism configuration (query, head, key/value axis splits) and its participating rank set, so that whole short sequences can share ranks with shards of long video sequences. The paper makes three main contributions: (1) a formal model of the placement problem and two theorems showing that disjoint-group placement (used by prior systems such as KnapFormer) suffers from a worst-case tradeoff between inter-group load imbalance and intra-group communication redundancy; (2) a two-stage planner consisting of Anchor Placement, which uses an exact compact formulation (ECF) of a CP-SAT model to balance compute-heavy sequences, and Filler Packing, which greedily places lighter whole sequences into remaining rank capacity; and (3) a Coalesced Attention Engine that efficiently executes mixed whole-sequence and distributed-attention-shard workloads. The evaluation on 16 A800 GPUs and 32 A6000 GPUs reports profile accuracy (3.4% MAPE on step makespan, 1.5% on memory), planner solve times of 33--119 ms with at most 0.32% modeled makespan overhead versus a joint-placement reference, and end-to-end speedups of 1.12--1.48x over KnapFormer on 16 A800 GPUs and 1.27--1.54x on 32 A6000 GPUs.

Significance. If the reported results are robust, Zellige is a meaningful step forward for mixed image-video DiT training. The measured speedups are substantial, and they come from a concrete, implementable system rather than from a purely theoretical construction. The theoretical results are self-contained and clearly stated, providing a clean characterization of the limitations of disjoint-group placement. The ECF formulation is a notable engineering contribution: it reduces planner solve times by orders of magnitude while preserving the exact in-model optima, as demonstrated by the ablation in Table III. The paper also reports measured end-to-end results on two different GPU testbeds, which strengthens the practical claims. The main weakness is that the validity of the planner's additive per-rank time model is demonstrated on only a narrow set of plan compositions, leaving some uncertainty about whether the headline speedups generalize to the full space of mixed workloads the system targets.

major comments (2)
  1. [IV-B and V-B/V-D] The additive per-rank time model of Section IV-B is validated only on the 21 plan compositions described in Section V-B, none of which match the composition of the headline 15-second-video workloads whose speedups are reported in Section V-D (Fig. 7). The 18 mixed validation plans contain exactly one 480p, one 720p, and one 1080p video plus a fixed set of 50 images, while the end-to-end evaluations sweep video-token share from 35% to 60% with 15-second videos. Because the planner's placement decisions depend entirely on the sum-of-prices model, unvalidated co-location patterns in the headline regimes could cause the planner to choose suboptimal placements, undermining the generality of the claimed speedups. The 'at most 0.32% overhead versus the joint-placement reference' in Section V-C compares modeled makespans only and therefore does not test model accuracy. The authors should add profile-accuracy measurements on plans drawn from the actual evaluation workloads, or explicitly restrict the generalizability claims to the validated composition space.
  2. [V-B and Eq. (1)] The validation of the hardware profile reports only the mean absolute percentage error on the predicted step makespan (3.4%) and peak memory (1.5%). However, the planner in Eq. (1) optimizes the maximum per-rank load, which requires the model to correctly rank alternative placements, not merely to predict absolute makespan accurately. A small MAPE on total makespan does not imply that the predicted ordering of candidate placements is correct, especially if errors are correlated across options. The paper does not report any rank-correlation or pairwise-preference accuracy, so the evidence that the planner's selection is reliable remains indirect. I recommend presenting, for a sample of candidate placements per workload, the correlation between predicted and measured makespans or the fraction of pairwise comparisons in which the model selects the truly faster placement.
minor comments (4)
  1. [I] In the Introduction, 'V AE compression' should be 'VAE compression' (the space between 'V' and 'AE' appears to be a formatting artifact).
  2. [Table III] The table's notation 'bTtwo/bTjoint' is not defined in the caption or table text; 'modeled makespans' should be written out clearly, and the speedup annotation in the mean column would be easier to read if it were a separate column.
  3. [IV-C, Eq. (8)] The Filler Packing score divides compute slack by F and memory slack by M_cap, which mixes dimensionless ratios with raw time and memory units; the heuristic is understandable, but a brief explanation of why this normalization is appropriate would improve clarity.
  4. [V-D] The end-to-end results are reported as point measurements without variance or repetition information; reporting the number of runs and standard deviations across seeds would make the speedup claims more robust.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the tradeoff theorems are proven from stated placement definitions, profiler prices are measured calibration inputs, and the 21-plan makespan comparison is a genuine held-out prediction.

full rationale

The central derivation is self-contained. Sec. III-C/III-D proves Theorems 1 and 2 directly from the definitions of disjoint-group and joint action spaces, with no reliance on external or prior-work results; the bounds are derived from the average-load lower bound and the construction of a one-giant-plus-fillers workload. The planner's cost model in Sec. IV-B sets A_{s,c,r}=P_{b(s),theta_c} using per-bucket, per-configuration prices that are measured on the hardware, and the Sec. V-B validation compares the additive sum against independently measured full-step makespans on 21 plan compositions, so the claim that the profiler 'predicts' makespan is an empirical, falsifiable model test rather than a fitted-input prediction. The ECF aggregation is an exact combinatorial reformulation of Eq. (1), not a new empirical claim. The end-to-end speedups in Sec. V-D are direct measured training-step comparisons against baseline systems, not outputs of the planner's own model. The self-citations ([16], [23]-[26]) support profiling practice, communication analysis, and memory modeling, but they are not load-bearing for the tradeoff theorems or the headline speedups; the memory model is additionally validated against measured peak memory. The '0.32% overhead versus the joint-placement reference' is a model-internal comparison, which limits what it demonstrates, but it is not circularity because it is not used as evidence that the model is accurate or that Zellige is fast in practice.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on measured profiler prices and on an additive time model that is empirically validated on a limited set of plan compositions. The theoretical theorems use a simplified ideal model. No new physical entities are introduced.

free parameters (4)
  • epsilon slack = 1e-3
    Chosen by hand as the allowed slack in the secondary objective of Anchor Placement (Eq. 7); a design choice, not a physical constant.
  • catalog cut k* = data-dependent
    Determined by minimizing within-set dispersion of log whole-sequence times (Eq. 4); a system hyperparameter that affects anchor/filler assignment.
  • profiled per-config price P_b,theta = measured on testbed
    Empirical measurements of per-rank execution time for each bucket-configuration pair; these are the core inputs to the planner and are fitted to the specific GPU hardware.
  • memory calibration M_base and M_usable = calibrated
    Calibrated against measured peak allocated memory to define the memory cap for Eq. (1); they are fitted to the implementation and hardware.
assumptions (4)
  • domain assumption Dense self-attention compute grows quadratically with sequence length L, while memory grows approximately linearly.
    Assumed throughout Section II and used to motivate the problem; referenced to FlashAttention [7].
  • ad hoc to paper The ideal compute-only model assumes zero communication cost and nonbinding memory.
    Stated in Section III-C before Theorem 1; restricts the theorem's applicability to a simplified setting.
  • ad hoc to paper Per-rank execution time of a placement is the sum of per-sequence, per-configuration profiled prices.
    Section IV-B states this additive model; validated on 21 plans but not proven generally.
  • domain assumption Legal parallelism configurations must satisfy executor-specific constraints, e.g., Ulysses degree divides the number of attention heads.
    Section IV-A defines legality constraints from the attention implementations used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zellige: Moldable Sequence Placement for Mixed Image-Video DiT Training." pith.science (2026). https://pith.science/paper/L7DEPYNX

@misc{pith2026260801150,
  author       = {Pith},
  title        = {Pith review of: Zellige: Moldable Sequence Placement for Mixed Image-Video DiT Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L7DEPYNX}},
  note         = {Machine review of arXiv:2608.01150}
}
abstract

High-quality video generation requires training Diffusion Transformers (DiTs) jointly on image and video data, posing a mixed-length sequence training problem across GPUs. Existing systems rely on data parallelism (DP), context parallelism (CP), or their combination; we model these designs as disjoint-group placement and prove that they face a fundamental tradeoff between inter-group load imbalance and intra-group communication redundancy. We present Zellige, a moldable sequence placement system that jointly selects each sequence's parallelism configuration and participating ranks. Zellige consists of three components: a hardware profiler that estimates the execution time and memory consumption of candidate placements, a two-stage planner that balances compute-heavy anchor sequences and packs lighter filler sequences into the remaining capacity, and a coalesced attention engine that efficiently executes whole sequences alongside distributed-attention shards. Across 21 plans, the hardware profile predicts step makespan and peak allocated memory with mean absolute percentage errors of $3.4%$ and $1.5%$, respectively. The two-stage planner solves each batch in 33--119 ms, significantly faster than a joint-placement reference that optimizes all sequences together, while their modeled makespans differ by at most $0.32%$. In end-to-end evaluations, Zellige outperforms KnapFormer by $1.12$--$1.48\times$ on 16 A800 GPUs and $1.27$--$1.54\times$ on 32 A6000 GPUs.

Figures

Figures reproduced from arXiv: 2608.01150 by the authors.

Figure 1
Figure 1. Token balance does not ensure compute balance. From [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Rank imbalance and CP communication cost. Panel (a) [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Zellige workflow. A catalog cut partitions the batch into [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: End-to-end step makespan on eight A800 GPUs. Each [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Measured resource breakdown for the high-load V50 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: End-to-end step makespan for six workloads with 15- [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 31 canonical work pages

  1. [1]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 6840–6851

    2020

  2. [2]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2022, pp. 10 684–10 695

    2022

  3. [3]

    Video diffusion models,

    J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 8633–8646

    work page 2022

  4. [4]

    Wan: Open and advanced large-scale video generative models,

    Team Wanet al., “Wan: Open and advanced large-scale video generative models,”arXiv preprint arXiv:2503.20314, 2025

    arXiv 2025

  5. [5]

    Hunyuanvideo: A systematic framework for large video generative models,

    W. Konget al., “Hunyuanvideo: A systematic framework for large video generative models,”arXiv preprint arXiv:2412.03603, 2024

    arXiv 2024

  6. [6]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 4195–4205

    work page 2023

  7. [7]

    FlashAttention: Fast and memory-efficient exact attention with IO-awareness,

    T. Dao, D. Y . Fu, S. Ermon, A. Rudra, and C. R ´e, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 16 344–16 359

    work page 2022

  8. [8]

    Efficient large-scale language model training on gpu clusters using megatron-lm,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on gpu clusters using megatron-lm,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and ...

    work page 2021

Show all 42 references
  1. [9]

    LoongTrain: Efficient training of long-sequence LLMs with head-context parallelism,

    D. Guet al., “LoongTrain: Efficient training of long-sequence LLMs with head-context parallelism,”arXiv preprint arXiv:2406.18485, 2024

    2024 arXiv

  2. [10]

    Hydraulis: Balancing large transformer model training via co-designing parallel strategies and data assignment,

    H. Li, F. Fu, S. Lin, H. Ge, X. Wang, J. Niu, J. Xue, Y . Tao, D. Wang, J. Jiang, and B. Cui, “Hydraulis: Balancing large transformer model training via co-designing parallel strategies and data assignment,” Proceedings of the ACM on Management of Data, vol. 3, no. 6, pp. 1–30, 2025

    2025

  3. [11]

    AdaptiveLoad: Towards efficient video diffusion transformer training,

    Y . Guo, Y . Guo, Z. Guan, H. Sun, W. Huang, W. Xu, J. Long, S. Di, and J. Xiong, “AdaptiveLoad: Towards efficient video diffusion transformer training,”arXiv preprint arXiv:2605.17923, 2026

    2026 arXiv

  4. [12]

    Usp: A unified sequence parallelism approach for long context generative ai,

    J. Fang and S. Zhao, “Usp: A unified sequence parallelism approach for long context generative ai,”arXiv preprint arXiv:2405.07719, 2024

    2024 arXiv

  5. [13]

    Knapformer: An online load balancer for efficient diffusion transformers training,

    K. Zhang, P. Wang, S. Bi, J. Zhang, and Y . Xiong, “Knapformer: An online load balancer for efficient diffusion transformers training,”arXiv preprint arXiv:2508.06001, 2025

    2025 arXiv

  6. [14]

    Efficient approximation algorithms for schedul- ing moldable tasks,

    X. Wu and P. Loiseau, “Efficient approximation algorithms for schedul- ing moldable tasks,”European Journal of Operational Research, vol. 310, no. 1, pp. 71–83, 2023

    2023

  7. [15]

    The CP-SAT-LP solver,

    L. Perron, F. Didier, and S. Gay, “The CP-SAT-LP solver,” inProc. International Conference on Principles and Practice of Constraint Pro- gramming (CP), ser. Leibniz International Proceedings in Informatics, vol. 280, 2023, pp. 3:1–3:2

    2023

  8. [16]

    KernelFlume: Elastic core-attention scaling for agentic long-context decoding,

    G. Xiang, X. Kang, L. Zhang, W. Lin, S. Shi, Y . Wang, and X. Chu, “KernelFlume: Elastic core-attention scaling for agentic long-context decoding,”arXiv preprint arXiv:2606.29207, 2026

    2026 arXiv

  9. [17]

    The Llama 3 herd of models,

    A. Grattafioriet al., “The Llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models,

    S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang, S. L. Song, S. Rajbhan- dari, and Y . He, “Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models,”arXiv preprint arXiv:2309.14509, 2023

    2023 arXiv

  11. [19]

    Ring attention with blockwise transformers for near-infinite context,

    H. Liu, M. Zaharia, and P. Abbeel, “Ring attention with blockwise transformers for near-infinite context,”arXiv preprint arXiv:2310.01889, 2023

    2023 arXiv

  12. [20]

    FlexSP: Accelerating large language model training via flexible sequence parallelism,

    Y . Wang, S. Wang, S. Zhu, F. Fu, X. Liu, X. Xiao, H. Li, J. Li, F. Wu, and B. Cui, “FlexSP: Accelerating large language model training via flexible sequence parallelism,” inProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages...

    2025

  13. [21]

    Performance modeling and evaluation of distributed deep learning frameworks on GPUs,

    S. Shi, Q. Wang, and X. Chu, “Performance modeling and evaluation of distributed deep learning frameworks on GPUs,” in2018 IEEE DASC/PiCom/DataCom/CyberSciTech, 2018, pp. 949–957

    2018

  14. [22]

    FSMoE: A flexible and scalable training system for sparse mixture- of-experts models,

    X. Pan, W. Lin, L. Zhang, S. Shi, Z. Tang, R. Wang, B. Li, and X. Chu, “FSMoE: A flexible and scalable training system for sparse mixture- of-experts models,” inProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operatin...

    2025

  15. [23]

    Compass: Dissecting communication and computation operators for efficient LLM training,

    G. Xiang, L. Zhang, H. Yu, X. Pan, S. Shi, and X. Chu, “Compass: Dissecting communication and computation operators for efficient LLM training,” inIEEE INFOCOM 2026 - IEEE Conference on Computer Communications, 2026, pp. 1–10

    2026

  16. [24]

    MG-WFBP: Merging gradients wisely for ef- ficient communication in distributed deep learning,

    S. Shi, X. Chu, and B. Li, “MG-WFBP: Merging gradients wisely for ef- ficient communication in distributed deep learning,”IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 8, pp. 1903–1917, 2021

    1903

  17. [25]

    DeAR: Acceler- ating distributed deep learning with fine-grained all-reduce pipelining,

    L. Zhang, S. Shi, X. Chu, W. Wang, B. Li, and C. Liu, “DeAR: Acceler- ating distributed deep learning with fine-grained all-reduce pipelining,” in2023 IEEE 43rd International Conference on Distributed Computing Systems (ICDCS), 2023, pp. 142–153

    2023

  18. [26]

    Xema: Efficient diffusion serving through fine-grained memory man- agement and auto-configuration,

    X. Kang, G. Xiang, S. Li, Y . Wang, S. Shi, L. Zhang, and X. Chu, “Xema: Efficient diffusion serving through fine-grained memory man- agement and auto-configuration,”arXiv preprint arXiv:2607.11136, 2026

    2026 arXiv

  19. [27]

    Reducing activation recomputation in large transformer models,

    V . A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models,” inProceedings of Machine Learning and Systems (MLSys), vol. 5, 2023, pp. 341–353

    2023

  20. [28]

    On grouping for maximum homogeneity,

    W. D. Fisher, “On grouping for maximum homogeneity,”Journal of the American Statistical Association, vol. 53, no. 284, pp. 789–798, 1958

    1958

  21. [29]

    On a bicriterion for- mulation of the problems of integrated system identification and system optimization,

    Y . Y . Haimes, L. S. Lasdon, and D. A. Wismer, “On a bicriterion for- mulation of the problems of integrated system identification and system optimization,”IEEE Transactions on Systems, Man, and Cybernetics, vol. SMC-1, no. 3, pp. 296–297, 1971

    1971

  22. [30]

    Exploiting orbits in symmetric ILP,

    F. Margot, “Exploiting orbits in symmetric ILP,”Mathematical Pro- gramming, vol. 98, no. 1–3, pp. 3–21, 2003

    2003

  23. [31]

    Patch n’ pack: NaViT, a vision transformer for any aspect ratio and resolution,

    M. Dehghaniet al., “Patch n’ pack: NaViT, a vision transformer for any aspect ratio and resolution,” inAdvances in Neural Information Processing Systems, vol. 36, 2023, pp. 2252–2274

    2023

  24. [32]

    Openvid-1m: A large-scale high-quality dataset for text-to-video generation,

    K. Nan, R. Xie, P. Zhou, T. Fan, Z. Yang, Z. Chen, X. Li, J. Yang, and Y . Tai, “Openvid-1m: A large-scale high-quality dataset for text-to-video generation,” inInternational Conference on Learning Representations, 2025

    2025

  25. [33]

    Koala-36m: A large- scale video dataset improving consistency between fine-grained condi- tions and video content,

    Q. Wang, Y . Shi, J. Ou, R. Chen, K. Lin, J. Wang, B. Jiang, H. Yang, M. Zheng, X. Tao, F. Yang, P. Wan, and D. Zhang, “Koala-36m: A large- scale video dataset improving consistency between fine-grained condi- tions and video content,” inProceedings of the IEEE/CVF Conference ...

    2025

  26. [34]

    DiffSynth-Studio,

    ModelScope Community, “DiffSynth-Studio,” https://github.com/ modelscope/DiffSynth-Studio, 2023, software repository, accessed 2026-07-28

    2023

  27. [35]

    xDiT: an inference engine for diffusion transformers (DiTs) with massive parallelism,

    J. Fang, J. Pan, X. Sun, A. Li, and J. Wang, “xDiT: an inference engine for diffusion transformers (DiTs) with massive parallelism,” arXiv preprint arXiv:2411.01738, 2024

    2024 arXiv

  28. [36]

    DSP: Dynamic sequence parallelism for multi-dimensional transform- ers,

    X. Zhao, S. Cheng, C. Chen, Z. Zheng, Z. Liu, Z. Yang, and Y . You, “DSP: Dynamic sequence parallelism for multi-dimensional transform- ers,” inProceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267. PMLR, ...

    2025

  29. [37]

    PipeDiT: Accelerating diffusion trans- formers in video generation with task pipelining and model decoupling,

    S. Wang, Q. Wang, and S. Shi, “PipeDiT: Accelerating diffusion trans- formers in video generation with task pipelining and model decoupling,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 12, 2026, pp. 10 092–10 100

    2026

  30. [38]

    Pulse: Training acceleration for large diffusion models with automatic pipeline parallelism,

    B. Sun, G. Jiang, L. Zhang, C. Chen, Y . Tao, Z. Che, J. Yu, S. Chang, H. Gu, F. Liu, and B. Li, “Pulse: Training acceleration for large diffusion models with automatic pipeline parallelism,”arXiv preprint arXiv:2606.19163, 2026

    2026 arXiv

  31. [39]

    Enabling parallelism hot switching for efficient training of large language models,

    H. Geet al., “Enabling parallelism hot switching for efficient training of large language models,” inProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, 2024, pp. 178–194

    2024

  32. [40]

    ByteScale: Communication-efficient scaling of LLM training with a 2048k context length on 16384 GPUs,

    H. Ge, J. Feng, Q. Huang, F. Fu, X. Nie, L. Zuo, H. Lin, B. Cui, and X. Liu, “ByteScale: Communication-efficient scaling of LLM training with a 2048k context length on 16384 GPUs,” inProceedings of the ACM SIGCOMM 2025 Conference, 2025, pp. 963–978

    2025

  33. [41]

    DCP: Addressing input dynamism in long-context training via dynamic context parallelism,

    C. Jiang, Z. Cai, Y . Tian, Z. Jia, Y . Wang, and C. Wu, “DCP: Addressing input dynamism in long-context training via dynamic context parallelism,” inProc. ACM SOSP, 2025, pp. 221–236

    2025

  34. [42]

    High-quality hypergraph partitioning,

    S. Schlag, T. Heuer, L. Gottesb ¨uren, Y . Akhremtsev, C. Schulz, and P. Sanders, “High-quality hypergraph partitioning,”ACM J. Exp. Algo- rithmics, vol. 27, pp. 1.9:1–1.9:39, 2022

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.