Pith. sign in

REVIEW 3 major objections 4 minor 100 references

LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This study measures which LLM serving frameworks and optimization methods appear in real open-source projects, finding vLLM dominant and parallel computation and memory management the most-used methods.

desk verdict The dataset and framework-level adoption findings are a real contribution, but the method-level headline is unsound: Multi-LoRA is systematically mislabeled as Network Pruning in Table 6, inflating one of the three top serving-method categories. read the letter →

arxiv 2608.03036 v1 pith:L3TXKR54 submitted 2026-08-04 cs.SE cs.AIcs.LG

Forough Majidi , Mohammad Mehdi Morovati , Foutse Khomh , Heng Li This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords LLMservingframeworksempiricalsoftwareengineeringGitHubrepositoryminingvefficientinferenceadoptionstudyopen-source
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a large-scale empirical study of how five LLM serving frameworks—vLLM, SGLang, TensorRT-LLM, LMDeploy, and FlashInfer—are actually used in open-source GitHub repositories. The authors aim to establish which frameworks and efficiency techniques developers really adopt, how they combine them, and what kinds of systems use them. They find that vLLM is both the most popular and the most adopted framework, while parallel computation, memory management, and network pruning are the most frequently observed serving-method categories. Multi-framework use is rare, but when frameworks appear together the pairings tend to join complementary capabilities such as vLLM's serving orchestration with FlashInfer's kernel-level attention optimizations. These results matter because they replace speculation with evidence about which serving innovations cross from research into practice.

What carries the argument

The central object is the method-to-API mapping: the authors read each framework's official documentation, assign its Python APIs and parameters to serving-method categories from an established taxonomy of efficient LLM serving, and then detect those APIs in GitHub Python files using abstract syntax tree analysis and regular expressions. Framework adoption is measured by counting unique repositories that import each framework's APIs, after filtering for popularity, activity, contributor count, and commit history. Repositories are further characterized by generating structured summaries of READMEs, about sections, and keywords with a language model, then clustering the summaries with embeddings, UMAP, HDBSCAN, and BERTopic to surface recurring intents, technical focuses, use cases, and system designs. The mapping is what turns raw API occurrences into the paper's method-adoption statistics.

What would settle it

Take a random sample of the 1,821 vLLM repositories, re-run the detection with Multi-LoRA and similar parameters assigned to their correct categories (low-rank adaptation, not network pruning), and compare the corrected network-pruning repository count with the reported 203; if the corrected count drops enough to move network pruning out of the top method categories, the headline ranking does not hold.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM serving in the wild is concentrated: one framework, vLLM, accounts for the bulk of observable adoption, and a small set of method categories—parallel computation (1,010 repositories), memory management (451), and network pruning (203)—dominate the rest. Adoption patterns are framework-specific, so FlashInfer repositories concentrate on kernel fusion and tailored attention, TensorRT-LLM appears mostly in GPU- and deployment-heavy contexts, and LMDeploy is the only framework observed with edge and resource-constrained models. When repositories combine frameworks, the recurring pairs (FlashInfer+vLLM, SGLang+vLLM, FlashInfer+SGLang) connect higher-level serving orchestration with lower-level attention and kernel work. The authors also claim the adopting repositories are heterogeneous in intent, spanning high-performance GPU serving, reinforcement-learning, retrieval-augmented generation, multimodal, and microservice-based systems.

Load-bearing premise

The paper's method-adoption results assume its documentation-derived mapping from framework APIs to serving-method categories is accurate, so a miscategorized parameter such as vLLM's Multi-LoRA being counted as network pruning could change the headline ranking of methods.

Editorial extensions

If this is right

  • Framework maintainers can use the observed method combinations to document and support common configurations rather than describing each optimization in isolation.
  • Practitioners choosing a serving stack can treat vLLM and LMDeploy as candidate frameworks for large and very large models, and LMDeploy as the one framework seen with edge and resource-constrained deployments.
  • Researchers studying serving efficiency should account for the fact that real systems combine methods such as memory management and parallel computation, rather than evaluating each technique alone.
  • Framework co-usage is rare enough that integration barriers, documentation gaps, or compatibility constraints deserve direct investigation as an explanation for single-framework reliance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method-category counts are only as trustworthy as the API-to-method mapping; a corrected mapping that treats vLLM's Multi-LoRA as low-rank adaptation rather than network pruning could shrink the reported network-pruning totals and should be checked before relying on the ranking.
  • The popularity-versus-adoption gap (FlashInfer ranks last in stars but third in adoption) suggests that GitHub star counts are a weak proxy for production relevance, a pattern likely to hold in other fast-moving AI infrastructure niches.
  • Because the study counts API imports rather than runtime behavior, a repository that imports a framework but does not actually exercise a method may overstate adoption; a targeted sample check on active import paths would test this.
  • The finding that vLLM appears in every repository-intent topic hints that a single general-purpose serving engine can cover application areas as different as RL training and multimodal pipelines, which could push future framework design toward even broader interfaces.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports an empirical study of how five LLM serving frameworks (vLLM, SGLang, TensorRT-LLM, LMDeploy, FlashInfer) are adopted in open-source GitHub repositories. The authors identify repositories via framework-specific imports and APIs, map detected APIs to an efficiency-method taxonomy adopted from Miao et al., and then analyze method adoption, within-framework method combinations, cross-framework co-usage, associations with model families/sizes/specializations, and repository-level topic clusters. The main claimed results are that vLLM is the most popular and most adopted framework; that parallel computation, memory management, and network pruning are the most frequently used serving-method categories; that multi-framework usage is limited; and that repositories using these frameworks span diverse intents, technical focuses, use cases, and system designs. A replication package is provided.

Significance. If the measurement pipeline were sound, the study would be a useful empirical map of LLM serving practice, with clear value for framework maintainers and practitioners. The paper has strengths: it uses an established taxonomy, reports a large corpus of repositories, and ships a replication package. However, the central quantitative finding on method adoption rests on an API-to-method mapping that is demonstrably wrong for at least one prominent case, and the cross-framework comparisons in the RQ3 analysis use raw counts that are dominated by sample-size differences. These issues affect the headline conclusions, not just peripheral details.

major comments (3)
  1. [§3.3, Table 6, Table 3] The method-to-API mapping is load-bearing and unvalidated. In Table 6, vLLM's 'Multi-LoRA' parameter is repeatedly assigned to the category 'Network Pruning' (e.g., in the FlashInfer+vLLM and SGLang+vLLM rows). LoRA is low-rank adaptation, not network pruning, and the paper itself discusses LoRA as a distinct serving capability in §4.1.3. Table 3 credits vLLM with 202 Network Pruning repositories (11% of 1,821), and this count is what places Network Pruning in the top three serving-method categories in the abstract and Finding 1. Section 7.1 does not report any precision, recall, or manual validation of the method-to-API classifier. Without recomputing all method counts with a corrected mapping, the headline method-ranking claim is unsupported.
  2. [§4.3.2, Table 13, Finding 3] The table compares frameworks by raw repository counts, yet framework sample sizes differ by two orders of magnitude (vLLM: 1,821; TensorRT-LLM: 15). The statements that vLLM appears in all identified topics and has the highest count in all of them are therefore largely artifacts of sample size, not evidence of broader relative adoption. The authors should normalize counts by framework repository total (or use appropriate statistical tests) before drawing conclusions about the breadth or concentration of each framework's adoption across repository intents, technical focuses, use cases, and system designs.
  3. [§4.2.2, Tables 4-6] The co-usage analysis inherits the same mapping error. In Table 6, Multi-LoRA is mapped to Network Pruning in a large share of the FlashInfer+vLLM and SGLang+vLLM rows, and the text in §4.2.2 reports 'Network Pruning appears in 14 repositories' as a pattern connecting vLLM with FlashInfer's attention/kernel methods. Because those counts are based on the erroneous mapping, the interpretive claim that these framework combinations connect complementary capabilities such as 'pruning' is not reliable. The co-usage counts and the qualitative interpretations built on them need to be redone after correcting the classification.
minor comments (4)
  1. [§3.1] The sentence 'we examine the top 100 results paper from each database' should be 'we examine the top 100 result papers from each database.'
  2. [Table 3] The 'Low-bit Quantization methods' row for LMDeploy shows an ambiguous double check mark and an unclear ratio cell; please clarify the notation so the reader can tell which methods are supported and what the percentages refer to.
  3. [§7.1] The description of adding three additional sub-subcategories would be more convincing if it reported the number of APIs affected by the additions and whether any inter-coder agreement was measured for the taxonomy mapping.
  4. [§4.2.1] There is a typo in the discussion paragraph ('seving' should be 'serving'), and several paragraphs in Section 4 repeat nearly identical phrasing; a thorough language edit is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: all findings are observational summaries of mined repository data, with the Multi-LoRA misclassification being a measurement-validity issue rather than circularity.

full rationale

This paper does not claim to derive serving-method adoption from a fitted model or to predict an outcome from its own inputs. Its chain is observational: it selects frameworks via literature and GitHub search (Section 3.1), builds a method-to-API mapping from official documentation (Section 3.3), mines Python repositories, and reports counts (Tables 2-3). The abstract's claim that parallel computation, memory management, and network pruning are the most frequent serving-method categories is a direct summary of Table 3 counts, not a quantity derived from the same counts by construction. The taxonomy is adopted from an external source (Miao et al. [3]); the added sub-subcategories (Section 4.1.3) are coding extensions, not equations that presuppose the result. Section 4.2's combination interpretations are explicitly labeled as possible interpretations and not claims of optimality. The method-to-API mapping in Table 6 does misclassify vLLM's Multi-LoRA as 'Network Pruning,' and this is a genuine measurement-validity threat to the headline Finding 1; however, a misclassification is an empirical error, not a circular step in which an output is equivalent to an input by definition. There is no load-bearing self-citation chain: framework documentation and prior external papers provide the API and taxonomy evidence. Accordingly, no circularity step can be exhibited under the required standard.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The study relies on several domain assumptions about taxonomy validity, API mapping correctness, framework representativeness, and search completeness. The API mapping assumption is demonstrably violated by the LoRA-to-pruning misclassification.

assumptions (4)
  • domain assumption The taxonomy of Miao et al. [3] is a valid and exhaustive classification of LLM serving methods.
    The paper adopts this taxonomy as a coding scheme without validating it against the actual methods implemented in the frameworks, and the authors add ad-hoc subcategories when methods do not fit.
  • domain assumption API import and parameter usage in Python scripts correctly indicates actual adoption of a serving method.
    The paper maps detected APIs to method categories in Section 3.3, but Table 6 shows a concrete misclassification (Multi-LoRA to Network Pruning), so this premise is not satisfied.
  • domain assumption The five selected frameworks are representative of the broader ecosystem of open-source LLM serving frameworks.
    Selection relies on GitHub stars and manual exclusion criteria (Section 3.1) that exclude large projects such as Ray, so findings may not generalize to all serving tools.
  • domain assumption A 500-byte file-size partitioning of the GitHub search space is complete enough to capture all relevant files.
    Section 3.3 states files above 500 KB are excluded based on preliminary investigation; the completeness of this partition is not verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs." pith.science (2026). https://pith.science/paper/L3TXKR54

@misc{pith2026260803036,
  author       = {Pith},
  title        = {Pith review of: LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L3TXKR54}},
  note         = {Machine review of arXiv:2608.03036}
}
read the original abstract

Large Language Models (LLMs) are integrated into software systems and AI services, making efficient LLM serving a concern for software engineering. Serving LLMs is challenging because inference requires computation, memory, GPU resources, and execution while maintaining latency and throughput. Although prior research has proposed LLM inference, optimization, and serving techniques and frameworks, little is known about how they are adopted in practice. In this study, we investigate the use of LLM serving frameworks and serving methods in open-source software systems. We identify and analyze five LLM-specific frameworks: vLLM, SGLang, TensorRT-LLM, LMDeploy, and FlashInfer. We examine how these frameworks and techniques are adopted individually and in combination, how adoption varies across categories of LLMs, and how repositories differ in intent, focus, use case, and architectural design. Our results show that vLLM is the most visible framework in popularity and adoption, while parallel computation, memory management, and network pruning are the most frequently used serving-method categories. Multi-framework usage is limited, suggesting that developers rely on a single serving framework; however, combined frameworks connect complementary capabilities across the serving stack. Framework adoption varies across model families, modalities, model sizes, domain specializations, and deployment settings. Repository-level analysis shows that LLM serving frameworks support applications and architectures, including Reinforcement Learning (RL)-based reasoning, multimodal generation and understanding, microservices, and cloud infrastructure. Overall, this study provides a large-scale empirical characterization of LLM serving framework adoption in practice and offers insights for researchers, framework maintainers, and practitioners working on LLM systems.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

100 extracted references · 50 canonical work pages

  1. [1]

    A survey of llm inference systems,

    J. Pan and G. Li, “A survey of llm inference systems,”arXiv preprint arXiv:2506.21901, 2025

    arXiv 2025

  2. [2]

    A survey on evaluation of large language models,

    Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wanget al., “A survey on evaluation of large language models,”ACM transactions on intelligent systems and technology, vol. 15, no. 3, pp. 1–45, 2024

    2024

  3. [3]

    Towards efficient generative large language model serving: A survey from algorithms to systems,

    X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia, “Towards efficient generative large language model serving: A survey from algorithms to systems,”ACM Computing Surveys, vol. 58, no. 1, pp. 1–37, 2025

    2025

  4. [4]

    Thunderserve: High-performance and cost-efficient llm serving in cloud environments,

    Y. Jiang, F. Fu, X. Yao, T. Wang, B. Cui, A. Klimovic, and E. Yoneki, “Thunderserve: High-performance and cost-efficient llm serving in cloud environments,”Proceedings of Machine Learning and Systems, vol. 7, 2025

    2025

  5. [5]

    Efficient memory management for large language model serving with page- dattention,

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with page- dattention,” inProceedings of the 29th symposium on operating systems principles, 2023, pp. 611–626

    2023

  6. [6]

    Sglang: Efficient execution of structured language model pro- grams,

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalezet al., “Sglang: Efficient execution of structured language model pro- grams,”Advances in neural information processing systems, vol. 37, pp. 62 557–62 583, 2024

    2024

  7. [7]

    Official document of tensorrt,

    TensorRT, “Official document of tensorrt,” https://developer.nvidia.com/tensorrt, ac- cessed: 2026-04-01. LLM Serving in the Wild 51

    2026

  8. [8]

    Official document of lmdeploy,

    LMDeploy, “Official document of lmdeploy,” https://github.com/InternLM/lmdeploy, accessed: 2026-04-01

    2026

Show all 100 references
  1. [9]

    Official document of flashinfer,

    FlashInfer, “Official document of flashinfer,” https://github.com/flashinfer- ai/flashinfer, accessed: 2026-04-01

    2026

  2. [10]

    Beyond efficiency: A systematic survey of resource-efficient large language mod- els,

    G. Bai, Z. Chai, C. Ling, S. Wang, J. Lu, N. Zhang, T. Shi, Z. Yu, M. Zhu, Y. Zhang et al., “Beyond efficiency: A systematic survey of resource-efficient large language mod- els,”arXiv preprint arXiv:2401.00625, 2024

    2024 arXiv

  3. [11]

    Efficient large language models: A survey,

    Z. Wan, X. Wang, C. Liu, S. Alam, Y. Zheng, J. Liu, Z. Qu, S. Yan, Y. Zhu, Q. Zhang et al., “Efficient large language models: A survey,”arXiv preprint arXiv:2312.03863, 2023

    2023 arXiv

  4. [12]

    A survey of resource-efficient llm and multimodal foundation models,

    M. Xu, W. Yin, D. Cai, R. Yi, D. Xu, Q. Wang, B. Wu, Y. Zhao, C. Yang, S. Wang et al., “A survey of resource-efficient llm and multimodal foundation models,”arXiv preprint arXiv:2401.08092, 2024

    2024 arXiv

  5. [14]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding,

    H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui, “Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding,”Findings of the Association for Computational Linguistics: ACL 2024, pp. 7655–7671, 2024

    2024

  6. [15]

    Model compression and efficient inference for large language models: A survey,

    W. Wang, W. Chen, Y. Luo, Y. Long, Z. Lin, L. Zhang, B. Lin, D. Cai, and X. He, “Model compression and efficient inference for large language models: A survey,”arXiv preprint arXiv:2402.09748, 2024

    2024 arXiv

  7. [16]

    A survey on model compression for large language models,

    X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang, “A survey on model compression for large language models,”Transactions of the Association for Computational Linguis- tics, vol. 12, pp. 1556–1577, 2024

    2024

  8. [17]

    Hybrid llm: Cost-efficient and quality-aware query routing,

    D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. Ruhle, L. V. Lakshmanan, and A. H. Awadallah, “Hybrid llm: Cost-efficient and quality-aware query routing,” arXiv preprint arXiv:2404.14618, 2024

    2024 arXiv

  9. [18]

    {Cost-Efficient}large language model serving for multi-turn conversations with{CachedAttention},

    B. Gao, Z. He, P. Sharma, Q. Kang, D. Jevdjic, J. Deng, X. Yang, Z. Yu, and P. Zuo, “{Cost-Efficient}large language model serving for multi-turn conversations with{CachedAttention},” in2024 USENIX annual technical conference (USENIX ATC 24), 2024, pp. 111–126

    2024

  10. [19]

    Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity,

    H. Xia, Z. Zheng, Y. Li, D. Zhuang, Z. Zhou, X. Qiu, Y. Li, W. Lin, and S. L. Song, “Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity,”arXiv preprint arXiv:2309.10285, 2023

    2023 arXiv

  11. [20]

    Evaluation of tools and frame- works for machine learning model serving,

    N. Beck, B. J. Stein, L. Helmer, and D. Wegener, “Evaluation of tools and frame- works for machine learning model serving,” in2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2025, pp. 80–90

    2025

  12. [21]

    Aibrix: Towards scalable, cost-effective large language model inference infrastructure,

    T. A. Team, J. Shan, V. Gupta, L. Xu, H. Shi, J. Zhang, N. Wang, L. Xu, R. Kang, T. Liuet al., “Aibrix: Towards scalable, cost-effective large language model inference infrastructure,”arXiv preprint arXiv:2504.03648, 2025

    2025 arXiv

  13. [22]

    Majidi, M

    F. Majidi, M. M. Morovati, and F. Khomh. (2026) Replication package of the paper. [Online]. Available: https://github.com/ForoughMajidi1995/LLM-Serving-Paper

    2026

  14. [23]

    What is artificial intelligence (ai)?

    IBM, “What is artificial intelligence (ai)?” https://www.ibm.com/think/topics/artificial- intelligence, 2026, accessed: 2026-04-30

    2026

  15. [24]

    S. J. Russell and P. Norvig,Artificial Intelligence: A Modern Approach, 4th ed. Pearson, 2020. [Online]. Available: http://aima.cs.berkeley.edu/

    2020

  16. [25]

    Defining intelligence: Bridging the gap between human and artificial perspectives,

    G. E. Gignac and E. T. Szodorai, “Defining intelligence: Bridging the gap between human and artificial perspectives,”Intelligence, vol. 104, p. 101832, 2024

    2024

  17. [26]

    Generative ai,

    S. Feuerriegel, J. Hartmann, C. Janiesch, and P. Zschech, “Generative ai,”Business & Information Systems Engineering, vol. 66, pp. 111–126, 2024

    2024

  18. [27]

    Generative artificial intelligence,

    L. Banh and G. Strobel, “Generative artificial intelligence,”Electronic Markets, vol. 33, p. 63, 2023

    2023

  19. [28]

    What is generative ai?

    IBM, “What is generative ai?” https://www.ibm.com/think/topics/generative-ai, 2026, accessed: 2026-04-30. 52 Forough Majidi et al

    2026

  20. [29]

    A comprehensive overview of large language models,

    H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian, “A comprehensive overview of large language models,”arXiv preprint arXiv:2307.06435, 2023

    2023 arXiv

  21. [30]

    What are large language models (llms)?

    IBM, “What are large language models (llms)?” https://www.ibm.com/think/topics/large-language-models, 2026, accessed: 2026- 04-30

    2026

  22. [31]

    Large language models meet nlp: A survey,

    L. Qin, Q. Chen, X. Feng, Y. Wu, Y. Zhang, Y. Li, M. Li, W. Che, and P. S. Yu, “Large language models meet nlp: A survey,”Frontiers of Computer Science, vol. 20, no. 11, p. 2011361, 2026

    2026

  23. [32]

    Large language models: A survey,

    S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,”arXiv preprint arXiv:2402.06196, 2024

    2024 arXiv

  24. [33]

    A survey on multimodal large language models,

    S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A survey on multimodal large language models,”National Science Review, vol. 11, no. 12, p. nwae403, 2024

    2024

  25. [34]

    A survey of state of the art large vision language models: Benchmark evaluations and challenges,

    Z. Li, X. Wu, H. Du, F. Liu, H. Nghiem, and G. Shi, “A survey of state of the art large vision language models: Benchmark evaluations and challenges,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2025, pp. 1587–1606

    2025

  26. [35]

    Large language models in medical and healthcare fields: applications, advances, and challenges,

    D. Wang and S. Zhang, “Large language models in medical and healthcare fields: applications, advances, and challenges,”Artificial intelligence review, vol. 57, no. 11, p. 299, 2024

    2024

  27. [37]

    A survey on large language model-empowered autonomous driving,

    Y. Zhu, S. Wang, W. Zhong, N. Shen, Y. Li, S. Wang, Z. Li, C. Wu, Z. He, and L. Li, “A survey on large language model-empowered autonomous driving,”arXiv preprint arXiv:2409.14165, 2024

    2024 arXiv

  28. [38]

    Actuarygpt: Applications of large language models to insurance and ac- tuarial work,

    C. Balona, “Actuarygpt: Applications of large language models to insurance and ac- tuarial work,”British Actuarial Journal, vol. 29, p. e15, 2024

    2024

  29. [39]

    A survey of large language models for financial applications: Progress, prospects and challenges,

    Y. Nie, Y. Kong, X. Dong, J. M. Mulvey, H. V. Poor, Q. Wen, and S. Zohren, “A survey of large language models for financial applications: Progress, prospects and challenges,”arXiv preprint arXiv:2406.11903, 2024

    2024 arXiv

  30. [40]

    A survey on efficient inference for large language models,

    Z. Zhou, X. Ning, K. Hong, T. Fu, J. Xu, S. Li, Y. Lou, L. Wang, Z. Yuan, X. Li et al., “A survey on efficient inference for large language models,”arXiv preprint arXiv:2404.14294, 2024

    2024 arXiv

  31. [41]

    An efficient model maintenance ap- proach for mlops,

    F. Majidi, F. Khomh, H. Li, and A. Nikanjam, “An efficient model maintenance ap- proach for mlops,”Empirical Software Engineering, vol. 31, no. 1, p. 21, 2026

    2026

  32. [42]

    Sprout: Green generative ai with carbon- efficient llm inference,

    B. Li, Y. Jiang, V. Gadepally, and D. Tiwari, “Sprout: Green generative ai with carbon- efficient llm inference,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 21 799–21 813

    2024

  33. [43]

    Measuring and improving the energy effi- ciency of large language models inference,

    M. F. Argerich and M. Pati˜ no-Mart´ ınez, “Measuring and improving the energy effi- ciency of large language models inference,”IEEE Access, vol. 12, pp. 80 194–80 207, 2024

    2024

  34. [44]

    An empirical study on the usage of au- tomated machine learning tools,

    F. Majidi, M. Openja, F. Khomh, and H. Li, “An empirical study on the usage of au- tomated machine learning tools,” in2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2022, pp. 59–70

    2022

  35. [45]

    Github rest api,

    G. developer guideline documentation, “Github rest api,” https://developer.github.com/v3/, 2021, accessed: 2025-09-05

    2021

  36. [46]

    Real faults in model context protocol (mcp) software: a comprehensive taxonomy,

    M. Taraghi, M. M. Morovati, and F. Khomh, “Real faults in model context protocol (mcp) software: a comprehensive taxonomy,”arXiv preprint arXiv:2603.05637, 2026

    2026 arXiv

  37. [47]

    Characterizing faults in agentic ai: A taxonomy of types, symptoms, and root causes,

    M. B. Shah, M. M. Morovati, M. M. Rahman, and F. Khomh, “Characterizing faults in agentic ai: A taxonomy of types, symptoms, and root causes,”arXiv preprint arXiv:2603.06847, 2026

    2026 arXiv

  38. [48]

    The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering,

    H. Li, H. Zhang, and A. E. Hassan, “The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering,”arXiv preprint arXiv:2507.15003, 2025

    2025 arXiv

  39. [49]

    Bug character- ization in machine learning-based systems,

    M. M. Morovati, A. Nikanjam, F. Tambon, F. Khomh, and Z. M. Jiang, “Bug character- ization in machine learning-based systems,”Empirical Software Engineering, vol. 29, no. 1, p. 14, 2024. LLM Serving in the Wild 53

    2024

  40. [50]

    Fault localization in deep learning- based software: A system-level approach,

    M. M. Morovati, A. Nikanjam, and F. Khomh, “Fault localization in deep learning- based software: A system-level approach,”arXiv preprint arXiv:2411.08172, 2024

    2024 arXiv

  41. [51]

    Bugs in machine learning- based systems: a faultload benchmark,

    M. M. Morovati, A. Nikanjam, F. Khomh, and Z. M. Jiang, “Bugs in machine learning- based systems: a faultload benchmark,”Empirical Software Engineering, vol. 28, no. 3, p. 62, 2023

    2023

  42. [52]

    Official document of vllm,

    vLLM, “Official document of vllm,” https://docs.vllm.ai/en/latest/, accessed: 2026- 04-01

    2026

  43. [53]

    Abstract syntax tree,

    Wikipedia, “Abstract syntax tree,” https://en.wikipedia.org/wiki/Abstract syntax - tree, accessed: 2026-04-28

    2026

  44. [54]

    About readmes,

    GitHub, “About readmes,” https://docs.github.com/en/repositories/managing-your- repositorys-settings-and-features/customizing-your-repository/about-readmes, 2026, accessed: 2026-04-28

    2026

  45. [55]

    Study the correlation between the readme file of github projects and their popularity,

    T. Wang, S. Wang, and T.-H. P. Chen, “Study the correlation between the readme file of github projects and their popularity,”Journal of Systems and Software, vol. 205, p. 111806, 2023

    2023

  46. [56]

    Classifying your repository with topics,

    GitHub, “Classifying your repository with topics,” https://docs.github.com/en/repositories/managing-your-repositorys-settings-and- features/customizing-your-repository/classifying-your-repository-with-topics, ac- cessed: 2026-04-28

    2026

  47. [57]

    (2026) Gpt-4o mini

    OpenAI. (2026) Gpt-4o mini. Accessed: 2026-03-18. [Online]. Available: https://developers.openai.com/api/docs/models/gpt-4o-mini

    2026

  48. [58]

    Generating github reposi- tory descriptions: A comparison of manual and automated approaches,

    J. Hellman, E. Jang, C. Treude, C. Huang, and J. L. Guo, “Generating github reposi- tory descriptions: A comparison of manual and automated approaches,”arXiv preprint arXiv:2110.13283, 2021

    2021 arXiv

  49. [59]

    Mining software reposito- ries for software architecture—a systematic mapping study,

    M. Soliman, M. Albonico, I. Malavolta, and A. Wortmann, “Mining software reposito- ries for software architecture—a systematic mapping study,”Information and Software Technology, vol. 181, p. 107677, 2025

    2025

  50. [60]

    Uncovering concerns of citizens through machine learning and social network sentiment analysis,

    S. Kumi, C. Snow, R. K. Lomotey, and R. Deters, “Uncovering concerns of citizens through machine learning and social network sentiment analysis,”Ieee Access, vol. 12, pp. 94 885–94 913, 2024

    2024

  51. [61]

    Prioritizing app reviews for developer responses on google play,

    M. Jafari, F. Majidi, and A. Heydarnoori, “Prioritizing app reviews for developer responses on google play,”arXiv preprint arXiv:2502.01520, 2025

    2025 arXiv

  52. [62]

    Sentence-bert: Sentence embeddings using siamese bert- networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert- networks,” inProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), 201...

    2019

  53. [63]

    Llm-assisted topic reduction for bertopic on social media data,

    W. Janssens, M. Bogaert, and D. V. d. Poel, “Llm-assisted topic reduction for bertopic on social media data,”arXiv preprint arXiv:2509.19365, 2025

    2025

  54. [64]

    Security vulnerabilities in ai-generated code: A large-scale analysis of public github repositories,

    M. Schreiber and P. Tippe, “Security vulnerabilities in ai-generated code: A large-scale analysis of public github repositories,” inInternational Conference on Information and Communications Security. Springer, 2025, pp. 153–172

    2025

  55. [65]

    An empirical study of blockchain repositories in github,

    A. Das, G. Uddin, and G. Ruhe, “An empirical study of blockchain repositories in github,” inProceedings of the 26th International Conference on Evaluation and As- sessment in Software Engineering, 2022, pp. 211–220

    2022

  56. [66]

    Ray: A distributed framework for emerging{AI}applica- tions,

    P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordanet al., “Ray: A distributed framework for emerging{AI}applica- tions,” in13th USENIX symposium on operating systems design and implementation (OSDI 18), 2018, pp. 561–577

    2018

  57. [67]

    Official document of openllm,

    OpenLLM, “Official document of openllm,” https://github.com/bentoml/OpenLLM, accessed: 2026-04-01

    2026

  58. [68]

    Official document of routellm,

    RouteLLM, “Official document of routellm,” https://github.com/lm-sys/RouteLLM, accessed: 2026-04-01

    2026

  59. [69]

    Official document of sglang,

    sglang, “Official document of sglang,” https://github.com/sgl-project/sglang, ac- cessed: 2026-04-01

    2026

  60. [70]

    Official document of ray,

    Ray, “Official document of ray,” https://github.com/ray-project/ray, accessed: 2026- 04-01

    2026

  61. [71]

    Official document of lightllm,

    LightLLM, “Official document of lightllm,” https://github.com/ModelTC/LightLLM, accessed: 2026-04-01. 54 Forough Majidi et al

    2026

  62. [72]

    Official document of fastdeploy,

    FastDeploy, “Official document of fastdeploy,” https://github.com/PaddlePaddle/FastDeploy, accessed: 2026-04-01

    2026

  63. [73]

    Flashinfer: Efficient and customizable attention engine for llm inference serving,

    Z. Ye, L. Chen, R. Lai, W. Lin, Y. Zhang, S. Wang, T. Chen, B. Kasikci, V. Grover, A. Krishnamurthyet al., “Flashinfer: Efficient and customizable attention engine for llm inference serving,”Proceedings of Machine Learning and Systems, vol. 7, 2025

    2025

  64. [74]

    Mell: Memory-efficient large language model serving via multi-gpu kv cache management,

    L. Qianli, H. Zicong, C. Fahao, L. Peng, and G. Song, “Mell: Memory-efficient large language model serving via multi-gpu kv cache management,”arXiv preprint arXiv:2501.06709, 2025

    2025 arXiv

  65. [75]

    A universal load balanc- ing principle and its application to large language model serving,

    Z. Chen, T. Bu, C. Song, X. Lu, Y. Ye, and Z. Zhou, “A universal load balanc- ing principle and its application to large language model serving,”arXiv preprint arXiv:2601.17855, 2026

    2026

  66. [76]

    Layerskip: Enabling early exit inference and self-speculative decoding,

    M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Romanet al., “Layerskip: Enabling early exit inference and self-speculative decoding,” inProceedings of the 62nd Annual Meeting of the Associa- tion for Computational Li...

    2024

  67. [77]

    Llm inference serving: Survey of recent advances and opportunities,

    B. Li, Y. Jiang, V. Gadepally, and D. Tiwari, “Llm inference serving: Survey of recent advances and opportunities,” in2024 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 2024, pp. 1–8

    2024

  68. [78]

    Efficiently scaling transformer inference,

    R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,”Proceedings of machine learning and systems, vol. 5, pp. 606–624, 2023

    2023

  69. [79]

    Systematic characterization of llm quantization: A performance, energy, and quality perspective,

    T. Shi and Y. Ding, “Systematic characterization of llm quantization: A performance, energy, and quality perspective,”arXiv preprint arXiv:2508.16712, 2025

    2025 arXiv

  70. [80]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,”Advances in neural information processing systems, vol. 36, pp. 10 088–10 115, 2023

    2023

  71. [81]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  72. [82]

    A survey on lora of large language models,

    Y. Mao, Y. Ge, Y. Fan, W. Xu, Y. Mi, Z. Hu, and Y. Gao, “A survey on lora of large language models,”Frontiers of Computer Science, vol. 19, no. 7, p. 197605, 2025

    2025

  73. [83]

    Reten- tive network: A successor to transformer for large language models,

    Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei, “Reten- tive network: A successor to transformer for large language models,”arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  74. [84]

    Rwkv: Reinventing rnns for the transformer era,

    B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, L. Derczynskiet al., “Rwkv: Reinventing rnns for the transformer era,” inFindings of the association for computational linguistics: EMNLP 2023, 2023, pp. 14 048–14 077

    2023

  75. [85]

    Non-autoregressive neural machine translation,

    J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher, “Non-autoregressive neural machine translation,”arXiv preprint arXiv:1711.02281, 2017

    2017 arXiv

  76. [86]

    Parallelism and scaling,

    vLLM, “Parallelism and scaling,” https://docs.vllm.ai/en/stable/serving/parallelism - scaling/, 2026, accessed: 2026-05-11

    2026

  77. [87]

    Performance and tuning: Chunked prefill,

    ——, “Performance and tuning: Chunked prefill,” https://docs.vllm.ai/en/v0.4.2/models/performance.html, 2026, accessed: 2026- 05-11

    2026

  78. [88]

    Quantization,

    H. Face, “Quantization,” https://huggingface.co/docs/optimum/en/concept - guides/quantization, accessed: 2026-05-11

    2026

  79. [89]

    Overview,

    N. TensorRT-LLM, “Overview,” https://nvidia.github.io/TensorRT- LLM/overview.html, 2026, accessed: 2026-05-1

    2026

  80. [90]

    Multi-head, multi-query, and group-query attention,

    ——, “Multi-head, multi-query, and group-query attention,” https://nvidia.github.io/TensorRT-LLM/advanced/gpt-attention.html, 2026, ac- cessed: 2026-05-1

    2026

  81. [91]

    Parallelism in tensorrt llm,

    ——, “Parallelism in tensorrt llm,” https://github.com/NVIDIA/TensorRT- LLM/blob/main/docs/source/features/parallel-strategy.md, 2026, accessed: 2026-05- 1

    2026

  82. [92]

    Quantization,

    ——, “Quantization,” https://nvidia.github.io/TensorRT- LLM/latest/features/quantization.html, accessed: 2026-05-11. LLM Serving in the Wild 55

    2026

  83. [93]

    FlexGen: High-throughput generative inference of large language models with a single GPU,

    Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Re, I. Stoica, and C. Zhang, “FlexGen: High-throughput generative inference of large language models with a single GPU,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedi...

    2023

  84. [94]

    DISP-LLM: Dimension-independent structural pruning for large language models,

    S. Gao, C.-H. Lin, T. Hua, Z. Tang, Y. Shen, H. Jin, and Y.-C. Hsu, “DISP-LLM: Dimension-independent structural pruning for large language models,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://openreview.net...

    2024

  85. [95]

    Challenges in deploying machine learning: A survey of case studies,

    A. Paleyes, R.-G. Urma, and N. D. Lawrence, “Challenges in deploying machine learning: A survey of case studies,”ACM Computing Surveys, vol. 55, no. 6, pp. 1–29,

  86. [96]

    A survey of machine learning in edge computing: Techniques, frameworks, applications, issues, and research directions,

    O. Jouini, K. Sethom, A. Namoun, N. Aljohani, M. H. Alanazi, and M. N. Alanazi, “A survey of machine learning in edge computing: Techniques, frameworks, applications, issues, and research directions,”Technologies, vol. 12, no. 6, p. 81, 2024. [Online]. Available: https://doi.o...

    2024 doi

  87. [97]

    Post-training large language models via reinforcement learning from self-feedback,

    C. van Niekerk, R. Vukovic, B. M. Ruppik, H.-c. Lin, and M. Gaˇ si´ c, “Post-training large language models via reinforcement learning from self-feedback,”arXiv preprint arXiv:2507.21931, 2025

    2025 arXiv

  88. [98]

    Reinforcement learning for reasoning in large language models with one training example, 2025,

    Y. Wang, Q. Yang, Z. Zeng, L. Ren, L. Liu, B. Peng, H. Cheng, X. He, K. Wang, J. Gaoet al., “Reinforcement learning for reasoning in large language models with one training example, 2025,”URL https://arxiv. org/abs/2504.20571, 2025

    2025 arXiv

  89. [99]

    Demystifying cost-efficiency in llm serving over heterogeneous gpus,

    Y. Jiang, F. Fu, X. Yao, G. He, X. Miao, A. Klimovic, B. Cui, B. Yuan, and E. Yoneki, “Demystifying cost-efficiency in llm serving over heterogeneous gpus,”arXiv preprint arXiv:2502.00722, 2025

    2025 arXiv

  90. [100]

    A study of llms’ preferences for libraries and programming languages,

    L. Twist, J. M. Zhang, M. Harman, D. Syme, J. Noppen, H. Yannakoudakis, and D. Nauck, “A study of llms’ preferences for libraries and programming languages,” inFindings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 331–351

    2026

  91. [101]

    The promises and perils of mining github,

    E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian, “The promises and perils of mining github,” inProceedings of the 11th Working Con- ference on Mining Software Repositories, ser. MSR 2014. New York, NY, USA: Association for Computing Machinery, 2...

    2014

  92. [2022]

    Available: https://doi.org/10.1145/3533378

    [Online]. Available: https://doi.org/10.1145/3533378

    doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.