Pith. sign in

REVIEW 5 minor 86 references

LobRA: Multi-tenant Fine-tuning over Heterogeneous Data

T0 review · 0 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Heterogeneous replica deployment and per-step workload-balanced dispatch cut joint LoRA fine-tuning GPU seconds by 45–60%.

desk verdict Solid systems paper: heterogeneous LoRA fine-tuning replicas with per-step workload-balanced dispatch cut GPU seconds 45-60% under a padding baseline; a packing baseline would shrink but not erase the gain. read the letter →

arxiv 2509.01193 v1 pith:NYKNB5KL submitted 2025-09-01 cs.DC

Sheng Lin , Fangcheng Fu , Haoyang Li , Hao Ge , Xuanyu Wang , Jiawen Niu , Yaofeng Tu , Bin Cui This is my paper · ORCID
classification cs.DC
keywords LoRAfine-tuningmulti-tenantheterogeneousmodelparallelismworkload-balanceddatadispatchingvariable-lengthsequencessequencepackingandpaddingGPUschedulingparameter-efficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the cost of jointly fine-tuning many LoRA adapters on one shared base model is dominated by two data-side mismatches, not by model size alone. First, different fine-tuning tasks have very different sequence lengths, so a single homogeneous replica configuration forces every replica to use more GPUs than short sequences need. Second, sequence-length distributions are skewed toward short texts, so dispatching each sequence to the cheapest capable replica leaves high-parallelism replicas idle. LobRA deploys heterogeneous fine-tuning replicas with different parallel configurations and, for every training step, dispatches data to balance their workloads, with bucket boundaries chosen dynamically to minimize padding. Experiments on 7B, 32B, and 70B models over 16–64 GPUs show GPU seconds for joint fine-tuning fall by 45.03%–60.67% relative to the homogeneous fused baseline.

What carries the argument

The key object is the heterogeneous FT replica: a copy of the shared base model plus its LoRA adapters, instantiated with a specific tensor- and pipeline-parallel configuration and therefore a specific maximum supportable sequence length and per-GPU throughput. LobRA combines these replicas with a per-step workload-balanced data dispatching scheme, formulated as minimizing the slowest replica's running time subject to every sequence being processed. A dynamic bucketing routine picks sequence-length bucket boundaries per batch by dynamic programming, so that padding is minimized and the dispatch problem has few variables. The two-stage decomposition of deployment planning and step-wise dispat

What would settle it

Run the same 7B and 70B workloads with a homogeneous fused baseline that uses sequence packing (block-diagonal causal masks) instead of padding, keeping batch sizes, models, and hardware fixed. If the packed baseline's GPU seconds fall within 20% of LobRA's, the headline speedup is an artifact of the padding baseline rather than of heterogeneous deployment itself.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that joint LoRA fine-tuning should be modeled as a min-max optimization over two coupled decisions: how many replicas to deploy with each parallel configuration, and which sequences to send to each replica at each step. Because the joint problem is too slow to solve every step, LobRA decomposes it: a deployment plan that is optimal in expectation is chosen once from the sequence-length distribution, and a fast integer-linear dispatch is solved per step for the actual batch. The empirical payoff is the GPU-second reduction; the authors also report that the decomposition tracks the joint optimum within 15% and the cost model tracks actual runt

Load-bearing premise

The reported savings assume that variable-length batches are padded to a common length rather than packed; the authors state the designs also apply to packing but do not evaluate that setting, so a packing-based baseline could reduce the 45–60% advantage.

Editorial extensions

If this is right

  • Fine-tuning service providers can serve concurrent LoRA requests on a shared base model with 45–60% fewer GPU seconds than the current fused-batch practice.
  • Larger base models benefit more: the savings grow from 45.03% on 7B to 60.67% on 70B, because heterogeneous replicas avoid the cross-server communication that a homogeneous high-parallel setting pays for all sequences.
  • Per-step dispatch adapts to the random composition of each batch, so the deployment stays fixed while the data routing changes; the planning overhead is hidden behind the previous step's training.
  • The same framework applies when sequence packing replaces padding, since the optimization only depends on the time-cost function being linear in the number of assigned sequences.
  • The solution quality remains close to the intractable joint optimum: the two-stage decomposition is within 15% of the original problem in estimated time, and the cost model within 10% of actual runtime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline speedup is measured against a padding-based homogeneous baseline; a packing-based baseline would likely narrow the gap, so the 45–60% range should be read as conditional on the padding assumption.
  • The workload-balancing principle should transfer to other frozen-backbone fine-tuning methods and to co-serving inference, wherever compute per sample varies.
  • A natural next test is whether the deployment plan can be re-optimized incrementally as tasks arrive and leave, without checkpoint-restart, by warm-starting the deployment solver from the current plan.
  • Dynamic bucketing could be reused in single-task fine-tuning and pre-training, where variable-length micro-batches create pipeline bubbles; the paper's own sequential experiments show those gains are task-dependent and sometimes negative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. LobRA is a multi-tenant fine-tuning framework for jointly training multiple LoRA adapters over a shared base model. The paper identifies two data-heterogeneity problems in this setting: sequence-length variation across tasks and skewness in the length distribution. Its two main ideas are (i) deploying heterogeneous FT replicas with different TP/PP configurations and GPU footprints, and (ii) per-step workload-balanced dispatch of sequences among these replicas, augmented by dynamic bucketing to reduce padding. The problem is decomposed into a one-time deployment-planning stage and a per-step ILP dispatch stage. The system is implemented on Hetu and evaluated on Llama2-7B, Qwen2.5-32B, and Llama2-70B over 16-64 GPUs, reporting a 45.03%-60.67% reduction in GPU seconds relative to a homogeneous Task-Fused baseline, with ablations and a planner-quality check.

Significance. If the reported results hold, LobRA addresses a practically important cost in model-as-a-service fine-tuning. The strengths of the paper are substantial: end-to-end measurements at multiple scales, ablations that isolate the contributions of heterogeneous replicas, workload balancing, and dynamic bucketing, a planner-quality experiment (Figure 10), an artifact link, and an unusually honest discussion of cases where the method does not help (e.g., the LobRA-Sequential degradation in Appendix B.2). The paper also clearly states its scope, including the padding assumption in §2.1 and the heuristic nature of configuration pruning. The packing-baseline concern raised in review is a legitimate external-validity limitation, but it is not an internal inconsistency; the central comparative claim is measured, and the core designs are not logically tied to padding alone.

minor comments (5)
  1. [Abstract; §2.1; §5.2, Figure 7] The headline 45.03%-60.67% reduction should be explicitly qualified as being measured in the padding regime. The paper states in §2.1 that sequence packing can be more efficient and that the proposed designs apply to packing, but no packing-based baseline is measured. Since dynamic bucketing (Eq. 4) is explicitly a padding-minimization device and the ablation attributes a nontrivial fraction of the gain to it, the current unqualified abstract and Section 5.2 wording overstates generality. Please add a qualifier and, ideally, a packing-based Task-Fused comparison or a discussion of the expected effect.
  2. [Appendix D, Eq. (12)] The ILP/MINLP statements in §4 rely on T being linear in the dispatch counts. Eq. (12) contains a max term over t(b_j,s_j) and t(r_j,s_j). The text should state explicitly that t(r_j,s_j) ≤ t(b_j,s_j), so the max term is a constant independent of d_j, making T linear. Without this clarification, the reader cannot verify the claimed ILP formulation of Eq. (3).
  3. [§4.2-4.3, Eqs. (1)-(3)] The objective uses d_{i,j}/p_i, which is generally fractional, while the per-replica dispatch counts must be integers. The paper does not describe how the implementation divides d_{i,j} sequences among the p_i replicas when d_{i,j} is not divisible by p_i. Please clarify the tie-breaking or rounding used in the implementation, since it affects the workload-balance claim.
  4. [Appendix A, Theorem 1] The proof of Theorem 1 depends on Assumption 1 and Observation 1, both of which are empirical assumptions rather than proven statements. The lower-bound filtering should be described more explicitly as a heuristic; Table 5 validates it on the tested workloads, but the current text does not flag the assumption's limited generality enough.
  5. [§5.2, Competitors] The evaluation compares against self-implemented Task-Fused and Task-Sequential baselines. Since related work includes an existing joint-FT system (mLoRA, [67]), a direct comparison with such a system, or at least a comment on why Task-Fused is representative, would strengthen external validity. The NeMo comparison in Appendix C is useful but only covers homogeneous training.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline speedup is a measured result; the fitted cost model is an optimizer, not the evidence.

full rationale

The paper's central claim is an empirical measurement: Figure 7 reports actual GPU seconds for LobRA versus Task-Fused/Task-Sequential, with reductions of 45.03%-60.67% over 100 training steps. The cost model in Appendix D is fitted to offline profiling data and used inside the optimizer to choose replica deployment and per-step dispatch, but the reported speedups are measured, not derived from the model. Dynamic bucketing (Eq. 4) explicitly minimizes padding under the paper's stated padding assumption; this is a genuine optimization objective rather than a predicted conclusion. The paper openly assumes padding and cites external work (Bai et al. [5]) for the padding/packing trade-off, so the choice is not smuggled in via self-citation. Self-citations to Hetu [33,38] and prior group works are implementation/background references and are not the evidence for the efficiency gain; Appendix C independently benchmarks LobRA against NVIDIA NeMo under identical homogeneous configurations, providing external grounding. The paper itself flags limitations: Assumption 1 in Appendix A is acknowledged not to always hold and is used only as a pruning heuristic, and footnote 16 notes that pipeline-bubble reduction is orthogonal. These are scope/robustness caveats, not circularity. The possible shrinkage of the reported gains under a packing-based baseline is an external-validity/benchmark-choice concern, not an internal circularity. No load-bearing step equates an input to an output by definition or by a self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claimed efficiency gain is an empirical benchmark, not a theoretical derivation. The planning components depend on a fitted cost model and several hand-chosen hyperparameters (R, U, threshold); these are disclosed but not derived from first principles. No new physical/mathematical entities are introduced.

free parameters (4)
  • Cost model coefficients (per-configuration time fitting) = Not disclosed in paper (curve fitted to offline profiling data, Appendix D)
    The time cost function T used in Equations (1)-(3) is fitted to profiling measurements for each parallel configuration. Deployment and dispatch plans depend on these fitted values.
  • Number of sequence buckets R = 16 (default, sensitivity tested 4-32)
    Chosen by hand and evaluated; affects granularity of length dispatch and dynamic bucketing.
  • Lower-bound filtering threshold = 15% (default)
    Hand-chosen threshold in configuration pruning; deployment plans whose estimated lower bound exceeds the current minimum by >15% are discarded.
  • Initial interval boundaries U for dynamic bucketing = Equal-length division starting at 256 (e.g., 256, 512, ...)
    Pre-defined by the authors; the DP in Eq. (4) selects R boundaries from these U intervals.
assumptions (4)
  • domain assumption Memory consumption of Transformer training is linear in the summed sequence length per chunk
    Used in §2.2 to justify why each parallel configuration supports a maximum sequence length and to determine r_i; cites [8,9,73].
  • ad hoc to paper The time cost function T is linear with respect to the number of dispatched sequences d_{i,j}
    Stated in the §2.2 footnote and Appendix D as a requirement for the ILP/MINLP formulations; the explicit formulas (Eq. 10-12) contain floor/remainder terms, so exact linearity only holds approximately for large counts.
  • ad hoc to paper Assumption 1 (Appendix A): ATB ordering of configurations at one sequence length reflects throughput ordering at all shorter lengths
    Used to prove Theorem 1 and justify the lower-bound filtering heuristic. The authors state it does not always hold in real-world scenarios, but use it only as a relative filter.
  • domain assumption Sequence padding rather than packing is used for variable-length data
    Adopted in §2.1 based on prior work showing comparable efficiency of padding and packing; the quantitative claim is evaluated under padding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LobRA: Multi-tenant Fine-tuning over Heterogeneous Data." pith.science (2026). https://pith.science/paper/NYKNB5KL

@misc{pith2026250901193,
  author       = {Pith},
  title        = {Pith review of: LobRA: Multi-tenant Fine-tuning over Heterogeneous Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NYKNB5KL}},
  note         = {Machine review of arXiv:2509.01193}
}
read the original abstract

With the breakthrough of Transformer-based pre-trained models, the demand for fine-tuning (FT) to adapt the base pre-trained models to downstream applications continues to grow, so it is essential for service providers to reduce the cost of processing FT requests. Low-rank adaption (LoRA) is a widely used FT technique that only trains small-scale adapters and keeps the base model unaltered, conveying the possibility of processing multiple FT tasks by jointly training different LoRA adapters with a shared base model. Nevertheless, through in-depth analysis, we reveal the efficiency of joint FT is dampened by two heterogeneity issues in the training data -- the sequence length variation and skewness. To tackle these issues, we develop LobRA, a brand new framework that supports processing multiple FT tasks by jointly training LoRA adapters. Two innovative designs are introduced. Firstly, LobRA deploys the FT replicas (i.e., model replicas for FT) with heterogeneous resource usages and parallel configurations, matching the diverse workloads caused by the sequence length variation. Secondly, for each training step, LobRA takes account of the sequence length skewness and dispatches the training data among the heterogeneous FT replicas to achieve workload balance. We conduct experiments to assess the performance of LobRA, validating that it significantly reduces the GPU seconds required for joint FT by 45.03%-60.67%.

Figures

Figures reproduced from arXiv: 2509.01193 by the authors.

Figure 1
Figure 1. An illustration of the fusion of different LoRA adapters. conveys the opportunity to share the same base model across mul￾tiple FT tasks rather than maintaining individual model replica(s) for each one. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Illustration of applying sequence padding and packing to variable-length data of one batch. Sequence padding uses the special token ⟨PAD⟩ to ensure sequences within the same chunk are the same length. Sequence packing concatenates sequences together and uses the block-diagonal casual masks to avoid cross-contamination. [76] considered fusing multiple LoRA adapters for joint FT. Nev￾ertheless, they carry out the join… view at source ↗
Figure 4
Figure 4. An example of 4 FT tasks with four different approaches, where (a) denotes fine-tuning the 4 tasks sequentially, whilst (b)-(d) present three different designs discussed in §3. We focus on the total GPU seconds required to run one training step for each task. (e) illustrates the inputs and decision variables of Equation (1) based on (d). example of four FT tasks in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Overview of LobRA. To start the joint FT task, given the base model and the sequence length distribution of the FT datasets, LobRA determines the deployment plan of FT replicas that minimizes the running time in expectation. During the FT process, for each training ste…
Figure 6
Figure 6. Figure 6: Illustration of dynamic bucketing. a notorious combinatorial optimization problem. Although libraries like SCIP [3] support solving MINLP problems, it is very time￾consuming, especially when there are many decision variables (i.e., 𝑑𝑖,𝑗 and 𝑝𝑖 ). Specifically, the numb…
Figure 8
Figure 8. Figure 8: Ablation Studies (7B model, 16 A100-40GB GPUs). task. The detailed descriptions of the FT datasets and the batch size settings are provided in Appendix B.1 [4]. By default, we consider 6 tasks for the 7B model and 12 tasks for the other two models. We use the Adam opti…
Figure 9
Figure 9. Figure 9: Case studies (7B model, 16 A100-40GB GPUs). Each bar represents one kind of FT replica(s). Top: The per-step time of each kind of FT replica(s). Bottom: The organization of dispatched data in terms of their sequence lengths for each kind of FT replica(s). 20 40 60 80 1…
Figure 10
Figure 10. Figure 10: Left: Time cost of solving the original problem (Equa￾tion (1)) vs. the two-stage planning (dynamic bucketing + solving Equation (3)). The horizontal dashed line indicates the average per￾step time. Right: Comparison of estimated running time for solv￾ing the original…
Figure 12
Figure 12. Figure 12: Impact of number of buckets (i.e., 𝑅) in dy￾namic bucketing to the per-step time and padding ratios (7B model, 16 A100-40GB GPUs). The per-step time is scaled by that with 4 buckets. Effectiveness of Configuration Planning. To assess the scalability of configuration p…
Figure 13
Figure 13. Figure 13: Illustration of the 1F1B pipeline parallel execution with variable-length inputs. Cells in light background color and dark background color represent forward and backward processes, respectively. The blue and green indicate two kinds of micro-batches that are differen…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

86 extracted references · 64 canonical work pages

  1. [1]

    Optimization with PuLP

    2009. Optimization with PuLP. https://coin-or.github.io/pulp/

    2009

  2. [2]

    The Ymir Proejct: Dataset and Workload

    2023. The Ymir Proejct: Dataset and Workload. https://sites.google.com/view/ ymir-project#h.dw77b5uw44tb

    2023

  3. [3]

    SCIP: Solving Constraint Integer Programs

    2024. SCIP: Solving Constraint Integer Programs. https://www.scipopt.org/

    2024

  4. [4]

    Full Version (with Appendix) of LobRA

    2025. Full Version (with Appendix) of LobRA. https://github.com/ccchengff/ LobRA/blob/main/LobRA_Full_Version_with_Appendix.pdf

    2025

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. LongAlign: A Recipe for Long Context Alignment of Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024 (EMNLP Findings, 2024) . 1376–1395

    2024

  6. [6]

    Weibo Cai, Shulin Yang, Gang Sun, Qiming Zhang, and Hongfang Yu. 2023. Adaptive load balancing for parameter servers in distributed machine learning over heterogeneous networks. ZTE Communications 21, 1 (2023), 72

    2023

  7. [7]

    Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krish- namurthy. 2024. Punica: Multi-Tenant LoRA Serving. In Proceedings of Machine Learning and Systems 2024 (MLSys 2024)

    work page 2024

  8. [8]

    Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In International Conference on Learning Representations 2024 (ICLR 2024)

    work page 2024

Show all 86 references
  1. [9]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Annual Conference on Neural Information Processing Systems 2022 (NeurIPS 2022)

    2022

  2. [10]

    DataBricks. 2025. Documents for Foundation Model Fine-tuning. https://docs. databricks.com/aws/en/large-language-models/foundation-model-training

    2025

  3. [11]

    Harm de Vries. 2023. In the long (context) run. https://www.harmdevries.com/ post/context-length/

    2023

  4. [12]

    Letian Deng and Yanru Zhao. 2023. Deep learning-based semantic feature extraction: A literature review and future directions. ZTE communications 21, 2 (2023), 11

    2023

  5. [13]

    Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Stefano Soatto. 2024. Fewer Truncations Improve Language Modeling. In International Conference on Machine Learning 2024 (ICML 2024)

    2024

  6. [14]

    Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020. Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. CoRR abs/2002.06305 (2020)

    2020 arXiv

  7. [15]

    Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayi- heng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2024. How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition. In Proceedings of the 62nd Annual Meetin...

    2024

  8. [16]

    Fei Du, Xin-Jian Ma, Jing-Ru Yang, Yi Liu, Chao-Ran Luo, Xue-Bin Wang, Hai-Ou Jiang, and Xiang Jing. 2024. A Survey of LLM Datasets: From Autoregressive Model to AI Chatbot. J. Comput. Sci. Technol. (2024)

    2024

  9. [17]

    Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding Back-Translation at Scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018) . 489–500

    2018

  10. [18]

    Shaoduo Gan, Xiangru Lian, Rui Wang, Jianbin Chang, Chengjun Liu, Hongmei Shi, Shengzhuo Zhang, Xianghong Li, Tengxu Sun, Jiawei Jiang, Binhang Yuan, Sen Yang, Ji Liu, and Ce Zhang. 2021. BAGUA: Scaling up Distributed Learning with System Relaxations. Proc. VLDB Endow. 15, 4 (...

    2021

  11. [19]

    Wensheng Gan, Shicheng Wan, and Philip S. Yu. 2023. Model-as-a-Service (MaaS): A Survey. CoRR abs/2311.05804 (2023)

    2023 arXiv

  12. [20]

    Lei Guan, Dong-Sheng Li, Jiye Liang, Wen-Jian Wang, Ke-shi Ge, and Xicheng Lu. 2024. Advances of Pipeline Model Parallelism for Deep Learning Training: An Overview. J. Comput. Sci. Technol. (2024)

    2024

  13. [21]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations 2022 (ICLR 2022)

    2022

  14. [22]

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. 2024. LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. In First Conference on Language Modeling (COLM 2024)

    2024

  15. [23]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Annual Conference on Neural Informat...

    2019

  16. [24]

    Insu Jang, Zhenning Yang, Zhen Zhang, Xin Jin, and Mosharaf Chowdhury

  17. [25]

    Zhihao Jia, Matei Zaharia, and Alex Aiken. 2019. Beyond Data and Model Parallelism for Deep Neural Networks. In Proceedings of Machine Learning and Systems 2019 (MLSys 2019)

    2019

  18. [26]

    Youhe Jiang, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, and Bin Cui. 2023. OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI 2023). 2142–2150

    2023

  19. [27]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Opti- mization. In 3rd International Conference on Learning Representations 2015 (ICLR 2015)

    2015

  20. [28]

    Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael An- dersch, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Reducing Activation Recomputation in Large Transformer Models. InProceedings of Machine Learning and Systems 2023 (MLSys 2023)

    2023

  21. [29]

    Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. 2021. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. CoRR abs/2107.02027 (2021)

    2021 arXiv

  22. [30]

    Achintya Kundu, Rhui Dih Lee, Laura Wynter, Raghu Kiran Ganti, and Mayank Mishra. 2024. Enhancing Training Efficiency Using Packing with Flash Attention. CoRR abs/2407.09105 (2024)

    2024 arXiv

  23. [31]

    Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng, Wanghu Chen, Mingjie Tang, and Jianguo Wang. 2024. GPTuner: A Manual- Reading Database Tuning System via GPT-Guided Bayesian Optimization. Proc. VLDB Endow. 17, 8 (2024), 1939–1952

    2024

  24. [32]

    Daiyi Li, Yaofeng Tu, Xiangsheng Zhou, Yangming Zhang, and Zongmin Ma

  25. [33]

    Haoyang Li, Fangcheng Fu, Hao Ge, Sheng Lin, Xuanyu Wang, Jiawen Niu, Xupeng Miao, and Bin Cui. 2025. Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations. CoRR abs/2504.20490 (2025)

    2025 arXiv

  26. [34]

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala

  27. [35]

    Yang Liu and Mirella Lapata. 2019. Text Summarization with Pretrained Encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP 2019). 3728–3738

    2019

  28. [36]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations 2019 (ICLR 2019)

    2019

  29. [37]

    Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. 2024. A Survey on LoRA of Large Language Models. CoRR abs/2407.11046 (2024)

    2024 arXiv

  30. [38]

    Xupeng Miao, Xiaonan Nie, Hailin Zhang, Tong Zhao, and Bin Cui. 2023. Hetu: a highly efficient automatic parallel distributed deep learning system. Sci. China Inf. Sci. 66 (2023)

    2023

  31. [39]

    Xupeng Miao, Gabriele Oliaro, Xinhao Cheng, Mengdi Wu, Colin Unger, and Zhihao Jia. 2024. FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning. CoRR abs/2402.18789 (2024)

    2024

  32. [40]

    Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. 2022. Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism. Proc. VLDB Endow. 16, 3 (2022), 470–479

    2022

  33. [41]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. 2019. PipeDream: generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM Symposium on Operating Systems Prin...

    2019

  34. [42]

    Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Za- haria. 2021. Memory-Efficient Pipeline-Parallel DNN Training. In International Conference on Machine Learning 2021 (ICML 2021) , Vol. 139. 7937–7947

    2021

  35. [43]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on GPU clus...

    2021

  36. [44]

    Xiaonan Nie, Yi Liu, Fangcheng Fu, Jinbao Xue, Dian Jiao, Xupeng Miao, Yangyu Tao, and Bin Cui. 2023. Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent. Proc. VLDB Endow. 16, 12 (2023), 3781–3794

    2023

  37. [45]

    NVIDIA. 2024. NeMo. https://github.com/NVIDIA/NeMo

    2024

  38. [46]

    NVIDIA. 2024. NVIDIA Collective Communications Library (NCCL). https: //developer.nvidia.com/nccl

    2024

  39. [47]

    OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. https: //openai.com/blog/chatgpt

    2022

  40. [48]

    OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023)

    2023 arXiv

  41. [49]

    OpenAI. 2024. OpenAI Platform: Fine-tuning. https://platform.openai.com/ docs/guides/fine-tuning/

    2024

  42. [50]

    Venkatesh Balavadhani Parthasarathy, Ahtsham Zafar, Aafaq Iqbal khan, and Arsalan Shahid. 2024. The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities. C...

    2024 arXiv

  43. [51]

    Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2020. Recipes for building an open-domain chatbot.CoRR abs/2004.13637 (2020)

    2020 arXiv

  44. [52]

    Gon- zalez, and Ion Stoica

    Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gon- zalez, and Ion Stoica. 2024. SLoRA: Scalable Serving of Thousands of LoRA Adapters. In Proceedings of Machine Learning and Sys...

    2024

  45. [53]

    Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis

    Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis. 2024. In-Context Pretraining: Language Modeling Beyond Document Boundaries. In International Conference on Learning Representations 2...

    2024

  46. [54]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. CoRR abs/1909.08053 (2019)

    2019 arXiv

  47. [55]

    Arjun Singh, Nikhil Pandey, Anup Shirgaonkar, Pavan Manoj, and Vijay Aski

  48. [56]

    Snowflake. 2025. Fine-tuning (Snowflake Cortex). https://docs.snowflake.com/ en/user-guide/snowflake-cortex/cortex-finetuning

    2025

  49. [57]

    Zijian Song, Wenhan Zhang, Lifang Deng, Jiandong Zhang, Kaigui Bian, and Bin Cui. 2024. MultiLoRA: Multi-Directional Low Rank Adaptation for Multi-Domain Recommendation. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM 2024)...

    2024

  50. [58]

    Jakub Tarnawski, Deepak Narayanan, and Amar Phanishayee. 2021. Piper: Multidimensional Planner for DNN Parallelization. In Annual Conference on Neural Information Processing Systems 2021 (NeurIPS 2021) . 24829–24840

    2021

  51. [59]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy F...

    2023 arXiv

  52. [60]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Annual Conference on Neural Information Processing Systems 2017 (NeurIPS 2017)

    2017

  53. [61]

    Smith, Iz Beltagy, and Hannaneh Hajishirzi

    Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khy- athi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. In Ann...

    2023

  54. [62]

    Yujie Wang, Youhe Jiang, Xupeng Miao, Fangcheng Fu, Shenhan Zhu, Xiaonan Nie, Yaofeng Tu, and Bin Cui. 2024. Improving Automatic Parallel Training via Balanced Memory Workload Optimization. IEEE Trans. Knowl. Data Eng. 36, 8 (2024), 3906–3920

    2024

  55. [63]

    Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang. 2023. MultiLoRA: Democratizing LoRA for Better Multi-Task Learning. CoRR abs/2311.11501 (2023)

    2023 arXiv

  56. [64]

    Xun Wu, Shaohan Huang, and Furu Wei. 2024. Mixture of LoRA Experts. In The Twelfth International Conference on Learning Representations 2024 (ICLR 2024)

    2024

  57. [65]

    Yifei Xia, Fangcheng Fu, Wentao Zhang Jiawei Jiang, and Bin Cui. 2024. Efficient Multi-task LLM Quantization and Serving for Multiple LoRA Adapters. InAnnual Conference on Neural Information Processing Systems 2024 (NeurIPS 2024)

    2024

  58. [66]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jin...

    2024 arXiv

  59. [67]

    Zhengmao Ye, Dengchun Li, Zetao Hu, Tingfeng Lan, Jian Sha, Sicong Zhang, Lei Duan, Jie Zuo, Hui Lu, Yuanchun Zhou, and Mingjie Tang. 2023. mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs. CoRR abs/2312.02515 (2023)

    2023 arXiv

  60. [68]

    Huangzhao Zhang, Kechi Zhang, Zhuo Li, Jia Li, Jia Li, Yongmin Li, Yunfei Zhao, Yuqi Zhu, Fang Liu, Ge Li, and Zhi Jin. 2024. Deep learning for code generation: a survey. Sci. China Inf. Sci. 67 (2024)

    2024

  61. [69]

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. In International conference on machine learning (ICML 2020) , Vol. 119. 11328–11339

    2020

  62. [70]

    Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation. In Proceedings of the 58th Annual Meeting of the Association...

    2020

  63. [71]

    Zhongping Zhang, Yin Jia, Yuehan Hou, and Xinlu Yu. 2024. Explicit Behavior Interaction with Heterogeneous Graph for Multi-behavior Recommendation. Data Sci. Eng. 9 (2024)

    2024

  64. [72]

    Zhen Zhang, Shuai Zheng, Yida Wang, Justin Chiu, George Karypis, Trishul Chilimbi, Mu Li, and Xin Jin. 2022. MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud. Proc. VLDB Endow. 16, 1 (2022), 37–50

    2022

  65. [73]

    Pinxue Zhao, Hailin Zhang, Fangcheng Fu, Xiaonan Nie, Qibin Liu, Fang Yang, Yuanbo Peng, Dian Jiao, Shuaipeng Li, Jinbao Xue, Yangyu Tao, and Bin Cui

  66. [74]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. PyTorch FSDP: Experiences...

    2023

  67. [75]

    Xing, Joseph E

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yan- ping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Automating Inter- and Intra-Operator Par- allelism for Distributed Deep Learning. In 16th U...

    2022

  68. [76]

    Ying Zheng, Lei Jiao, Han Yang, Lulu Chen, Ying Liu, Yuxiao Wang, Yuedong Xu, Xin Wang, and Zongpeng Li. 2024. Online Scheduling and Pricing for Multi-LoRA Fine-Tuning Tasks. In Proceedings of the 53rd International Conference on Parallel Processing, (ICPP 2024). 357–366

    2024

  69. [77]

    Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. 2024. Multi-LoRA Composition for Image Generation. CoRR abs/2402.16843 (2024)

    2024 arXiv

  70. [78]

    Xuanhe Zhou, Guoliang Li, Zhaoyan Sun, Zhiyuan Liu, Weize Chen, Jianming Wu, Jiesi Liu, Ruohang Feng, and Guoyang Zeng. 2024. D-Bot: Database Diagnosis System using Large Language Models. Proc. VLDB Endow. 17, 10 (2024), 2514– 2527

    2024

  71. [79]

    Xuanhe Zhou, Zhaoyan Sun, and Guoliang Li. 2024. DB-GPT: Large Language Model Meets Database. Data Sci. Eng. 9 (2024)

    2024

  72. [80]

    Zhe Zhou, Xuechao Wei, Jiejing Zhang, and Guangyu Sun. 2022. PetS: A Unified Framework for Parameter-Efficient Transformers Serving. In2022 USENIX Annual Technical Conference (ATC 2022). 489–504

    2022

  73. [81]

    ✗” indicates the parallel configuration does not support processing the corresponding sequence length due to out-of-memory errors. The symbol “-

    Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis. In Findings of the Association for Computational Linguistics: NAACL (NAA...

    2024

  74. [2020]

    PyTorch Distributed: Experiences on Accelerating Data Parallel Training. Proc. VLDB Endow. 13, 12 (2020), 3005–3018

    2020

  75. [2022]

    ZTE Communications 20, S1 (2022), 27

    End-to-end chinese entity recognition based on bert-bilstm-att-crf. ZTE Communications 20, S1 (2022), 27

    2022

  76. [2023]

    In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP 2023)

    Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP 2023). 382–395

    2023

  77. [2024]

    CoRR abs/2406.02290 (2024)

    A Study of Optimizations for Fine-tuning Large Language Models. CoRR abs/2406.02290 (2024)

    2024 arXiv

  78. [2025]

    In Proceedings of the 2025 ACM International Conference on Management of Data (SIGMOD 2025)

    Efficiently Training 7B LLM with 1 Million Sequence Length on 8 GPUs. In Proceedings of the 2025 ACM International Conference on Management of Data (SIGMOD 2025)

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.