Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Context-Aware CodeLLM Eviction for AI-assisted Coding

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A context-aware eviction policy cuts model reloads and latency for self-hosted coding LLMs.

desk verdict A sensible multi-factor eviction idea whose headline results depend on oracle access to the future request queue; the paper needs a no-lookahead re-evaluation before its numbers can be trusted. read the letter →

arxiv 2506.18796 v1 pith:ZU3ODJ2K submitted 2025-06-23 cs.SE

Kishanthan Thangarajah , Boyuan Chen , Shi Chang , Ahmed E. Hassan This is my paper
classification cs.SE
keywords CodeLLMmodelevictionmultiplexingself-hostedservinglatencycachehitratecompletion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that when enterprises self-host many coding-specific large language models on limited accelerator memory, the standard least-recently-used eviction policy causes unnecessary model reloads and slow responses. It introduces CACE, a context-aware eviction policy that scores each resident model using four factors: recency, reload cost, predicted future demand from the request queue, and task criticality estimated by expected output length. In simulated IDE workloads, CACE substantially reduces the number of model evictions, raises cache hit rates, and cuts both time-to-first-token and end-to-end latency compared with LRU. The authors' central claim is that a multi-factor, task-aware eviction policy is practical and markedly better than recency-only policies for CodeLLM serving.

What carries the argument

The central object is the CACE eviction score, a weighted sum of four factors: P1 recency, P2 reloading cost, P3 future demand from the request queue, and P4 task criticality. The policy selects the model with the highest score for eviction, which protects recently used, slow-to-reload, soon-to-be-needed, and latency-critical models. The future-demand term uses a sliding window over the incoming request queue, and task criticality uses expected output token count, with code completion assumed to be TTFT-sensitive and code reasoning E2E-sensitive.

What would settle it

A controlled experiment in which the same CACE policy is run without access to any future request information (for example, using only historical demand or a trained predictor) and compared against LRU: if the cache hit rate and latency benefits shrink to near the LRU baseline under realistic online conditions, the central claim that CACE's context-aware factors, especially future demand, deliver practical gains would be falsified.

Watch

Extended reading notes

Core claim

CACE reduces model evictions, Time-to-First-Token (TTFT), and end-to-end (E2E) latency in self-hosted CodeLLM serving compared to state-of-the-art LRU-based multiplexing systems. The eviction score combines four terms: recency of use, model reload time, position in a sliding window of future requests, and expected output token count as a proxy for task criticality. In experiments with three workload patterns mixing code completion and code reasoning tasks across eight programming languages, CACE increased model cache hit rate to roughly 0.85-0.86, reduced model evictions by about 55%, lowered mean TTFT by up to 70%, and reduced E2E latency by up to 37% on average and 59% at the 99th percentile relative to LRU. Ablation studies show that task criticality and future demand contribute the most, while recency and reload cost have smaller effects.

Load-bearing premise

The eviction policy is evaluated with full knowledge of the future request queue, so the reported benefit of the future-demand factor assumes that upcoming requests can be seen or accurately predicted at eviction time.

Editorial extensions

If this is right

  • If CACE is correct, self-hosted CodeLLM serving stacks can support more models per accelerator by reducing cold-start reloads, improving interactive code completion responsiveness.
  • Eviction policies for LLM serving should consider task-level latency requirements and expected output length, not just access recency, to preserve interactive quality.
  • The finding that future demand and task criticality dominate suggests that lightweight request-queue-aware policies can outperform pure LRU without complex scheduling.
  • The complementary claims about combining CACE with KV-cache paging and CPU memory caching point toward a layered memory hierarchy for LLM serving.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The future-demand factor P3 is evaluated with full knowledge of the request stream, so in real deployments its benefit depends on how well future requests can be predicted; without a predictor, P3 would need to be estimated from historical patterns or replaced by a probabilistic model.
  • The ablation results suggest that a simpler policy using only task criticality and future demand may capture most of the gains, which could be tested directly as a two-factor variant.
  • The use of expected output tokens as a proxy for task criticality is coarse: real completions vary in length, and a more direct latency-sensitivity label or user intent signal could improve decisions.
  • Extending CACE with a CPU-memory cache tier as the authors suggest would shift the tradeoff: reload cost would drop, potentially changing the optimal weights for the P2 term.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CACE, a context-aware model eviction policy for self-hosted CodeLLM serving systems. CACE computes an eviction score for each resident model using four factors: recency (P1), reloading cost (P2), future demand inferred from the request queue (P3), and task criticality based on expected output length (P4). The authors evaluate CACE against an LRU baseline on three synthetic workload patterns (Uniform, IDE Heavy, and Language Popularity Skewed) with 16 CodeLLMs, reporting up to 70% lower mean TTFT, up to 37% lower mean E2E latency, higher cache hit rates, and fewer evictions. An ablation study (RQ3) isolates the contribution of each factor.

Significance. If the reported gains are realizable in an online setting, CACE would be a practical improvement over recency-only eviction for model multiplexing in enterprise coding assistants. The paper's ablations are a useful step toward understanding which factors matter in eviction decisions, and the authors are explicit about several threats to validity. However, the central evaluation appears to give CACE access to the ground-truth future request stream, so the headline results are currently an oracle bound rather than a demonstrated practical improvement.

major comments (3)
  1. [Section IV-B and Section V-A] The P3 'Future Demand' factor is defined in Section IV-B as i/w, where i is the model's position in L2 ('Models to be loaded'). Section V-A states that the entire synthetic request stream is shuffled before dispatch. This means that at every eviction decision, L2 can contain the complete remaining request sequence, giving CACE knowledge of the true future. In a real deployment, L2 can contain only requests that have already arrived and are queued. The RQ3 ablation shows that disabling P3 reduces Model Cache Hit Rate by 22% and increases P95 TTFT by 15s, so the reported TTFT/E2E gains and eviction reductions depend substantially on this lookahead. Please either evaluate CACE with L2 restricted to requests that have already arrived (or with a realistic forecast), or explicitly reframe the current results as an oracle upper bound and compare them against a no-lookahead variant to establish the practically achievable benefit.
  2. [Section IV-B, Algorithm 1] The eviction score formula uses the sliding-window length w and the task-criticality weight w1, but the paper never reports the values used in the experiments. The scaling constant 100 in the P2 term is also unreported. Without these values, the experiments cannot be reproduced. Please state the exact configuration for all reported results, including any tuning procedure.
  3. [Section V-A and Figures 2-9] The workloads are generated using a Poisson arrival process, which is stochastic, yet all reported results appear to come from a single run with no error bars, confidence intervals, or repeated seeds. Claims such as 'reduces TTFT by up to 70%' need to be supported by means and variances over multiple runs to rule out random variation and to allow readers to assess the stability of the improvements.
minor comments (4)
  1. [Section IV-B] The variable t is defined as 'last used time of the model in L1 (timestamp)', but the formula uses ln(t) in a reciprocal. If t is an absolute timestamp, then a more recently used model has a larger t, not a smaller one, which contradicts the stated rationale that smaller t should yield a larger inverse score contribution. Please clarify whether t is the time since last access (age) or a timestamp, and adjust the definition and formula accordingly.
  2. [Section V-A] The sentence 'We used Ray Serve [30] as the baseline system, which implements an LRU-based model multiplexing strategy, similar to RayServe and FaaSwap' is redundant and slightly confusing; please rephrase to state which baselines were actually evaluated.
  3. [Section V-A] Please specify the Poisson arrival rate used in the three workload patterns and explain how the shuffle interacts with the arrival timestamps, since the current description makes it unclear whether L2 is populated with all remaining requests at time zero or only with requests that have arrived by each decision point.
  4. [Throughout] There are minor typos, such as 'Reloading Cos' instead of 'Reloading Cost' in the RQ1 findings and 'shows the effectiveness only by considering' missing a word. A careful proofreading pass would improve clarity.

Circularity Check

1 steps flagged · score 7.0 of 10

CACE's P3 'Future Demand' term is computed from the test workload's actual future request queue (i/w), so the headline TTFT/E2E and eviction gains are partly an oracle artifact rather than a demonstrated prediction.

  1. self definitional [Section IV-B (Eviction Score Calculation, P3 definition), Section V-A (Case Study Setup), Section V-D/RQ3 (ablation)]
    "P3: Future Demand (Sliding Window Forecast) This factor estimates the urgency of future model usage based on the position of the model in the future request window. It is calculated as the fraction between model's i(index position of the model in L2) and the w(which is the sliding window length). ... Before dispatching the stream to the serving stack, we shuffle all requests to avoid bursty language or task-specific clusters."

    The eviction score's 'forecast' is defined as i/w, where i is the model's index in L2 ('models to be loaded'). In the evaluation, the entire request stream is generated and shuffled offline before dispatch, so L2 can contain the ground-truth future request sequence at every eviction decision. Thus P3 is not a prediction; it is read from the exact test workload that the system is supposed to anticipate. RQ3 shows this oracle factor is the second-most impactful (removing it lowers Model Cache Hit Rate by 22% and raises P95 TTFT by 15s), so the reported TTFT/E2E and eviction reductions are substantially attributable to information LRU cannot have. No no-lookahead variant is evaluated.

full rationale

The central derivation is not built on a self-citation chain: the only self-reference is the replication-package pointer [34], which is not load-bearing. The circularity is in the P3 factor. Section IV-B defines Future Demand as i/w, using the model's index position in L2, and Section V-A pre-generates and shuffles the whole 30-second request stream before dispatch, so the 'future request window' can be the actual remaining workload. Consequently, CACE's eviction score is partly a function of the ground-truth future accesses it claims to forecast, and the ablation attributes substantial benefit to this factor. The other factors (recency, reload cost, expected output length) are legitimate non-circular inputs, and the paper does not evaluate a no-lookahead version; the magnitude of the reported gains over LRU is therefore not established as a realistic prediction. This is a partial, load-bearing circularity rather than full equivalence.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on an ad hoc eviction score with unreported parameters, and on a domain assumption that the request queue provides reliable future-demand information. The evaluation further grants the policy oracle access to the test workload's future, which is the main load-bearing assumption.

free parameters (3)
  • w1 (task criticality weight)
    Defined in Section IV-B as 'weighting factor for output tokens'; no value or tuning procedure is reported, yet it directly scales P4 in the eviction score.
  • w (sliding window length)
    Used in P3 (i/w) and Algorithm 1; no value reported for experiments, and it controls how much future demand influences eviction.
  • Scaling constant 100 in load-time term
    The 1/(1 + l/100) term uses an arbitrary constant that sets the load-time scale; no sensitivity analysis is given.
assumptions (5)
  • domain assumption Model load time is a valid proxy for eviction and reload cost.
    Section IV-B P2 uses load time l to decide eviction desirability; assumes load time dominates other costs such as transfer bandwidth and disk contention.
  • domain assumption Expected output token count is a valid proxy for task latency criticality.
    Section IV-B P4 assumes shorter outputs correspond to latency-sensitive tasks like completion and longer outputs to reasoning; real tasks vary.
  • domain assumption The orchestrator can observe the incoming request queue to anticipate future demand.
    Section IV-B P3 uses index i in L2; in the synthetic evaluation the entire shuffled request stream is known before dispatch (Section V-A), so this becomes an oracle assumption.
  • ad hoc to paper Poisson arrivals over 30 seconds with shuffled requests represent realistic enterprise developer workloads.
    Section V-A defines the workload generation; no validation against real IDE telemetry, and shuffling destroys arrival-time structure.
  • domain assumption Single active instance per model and one accelerator per model instance.
    Section V-A and Threats to Validity state this; real deployments may replicate models, changing eviction dynamics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Context-Aware CodeLLM Eviction for AI-assisted Coding." pith.science (2026). https://pith.science/paper/ZU3ODJ2K

@misc{pith2026250618796,
  author       = {Pith},
  title        = {Pith review of: Context-Aware CodeLLM Eviction for AI-assisted Coding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZU3ODJ2K}},
  note         = {Machine review of arXiv:2506.18796}
}
read the original abstract

AI-assisted coding tools powered by Code Large Language Models (CodeLLMs) are increasingly integrated into modern software development workflows. To address concerns around privacy, latency, and model customization, many enterprises opt to self-host these models. However, the diversity and growing number of CodeLLMs, coupled with limited accelerator memory, introduce practical challenges in model management and serving efficiency. This paper presents CACE, a novel context-aware model eviction strategy designed specifically to optimize self-hosted CodeLLM serving under resource constraints. Unlike traditional eviction strategies based solely on recency (e.g., Least Recently Used), CACE leverages multiple context-aware factors, including model load time, task-specific latency sensitivity, expected output length, and recent usage and future demand tracked through a sliding window. We evaluate CACE using realistic workloads that include both latency-sensitive code completion and throughput-intensive code reasoning tasks. Our experiments show that CACE reduces Time-to-First-Token (TTFT) and end-to-end (E2E) latency, while significantly lowering the number of model evictions compared to state-of-the-art systems. Ablation studies further demonstrate the importance of multi-factor eviction in balancing responsiveness and resource efficiency. This work contributes practical strategies for deploying scalable, low-latency AI coding assistants in real-world software engineering environments.

Figures

Figures reproduced from arXiv: 2506.18796 by the authors.

Figure 1
Figure 1. The Overall Workflow of CACE The serving pipeline, as described in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Comparing Model Load Overhead against different request load [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 2
Figure 2. Comparing Model Cache Hit Rate against different request load [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Comparing E2E latency against different request load patterns of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparing Model Cache Hit Rate against different request load [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 4
Figure 4. Figure 4: Comparing TTFT against different request load patterns of CACE [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 8
Figure 8. Figure 8: Comparing E2E latency of different variants of CACE. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Comparing Model Cache Hit Rate of different variants of CACE. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 7
Figure 7. Figure 7: Comparing TTFT of different variants of CACE. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agentic Software Engineering: Foundational Pillars and a Research Roadmap

    cs.SE 2025-09 conditional novelty 5.0 of 10

    SASE introduces a dual-modality framework (SE for Humans and SE for Agents) with workbenches and structured artifacts to turn agentic coding into a disciplined engineering practice.

Reference graph

Works this paper leans on

56 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [1]

    A survey on large language models for code generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,”arXiv preprint arXiv:2406.00515, 2024

    arXiv 2024

  2. [2]

    A systematic evaluation of large language models of code,

    F. F. Xu, U. Alon, G. Neubig, and V . J. Hellendoorn, “A systematic evaluation of large language models of code,” inProceedings of the 6th ACM SIGPLAN international symposium on machine programming, 2022, pp. 1–10

    2022

  3. [3]

    Measuring github copilot’s impact on productivity,

    A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, “Measuring github copilot’s impact on productivity,”Communications of the ACM, vol. 67, no. 3, pp. 54–63, 2024

    2024

  4. [4]

    Cursor: The ai code editor,

    Cursor, “Cursor: The ai code editor,” 2024, accessed: 2025-05-20. [Online]. Available: https://www.cursor.com/

    work page 2024

  5. [5]

    Github copilot: The ai editor for everyone,

    G. Copilot, “Github copilot: The ai editor for everyone,” 2024, accessed: 2025-05-20. [Online]. Available: https://github.com/features/copilot

    work page 2024

  6. [6]

    Ai-assisted code author- ing at scale: Fine-tuning, deploying, and mixed methods evaluation,

    V . Murali, C. Maddila, I. Ahmad, M. Bolin, D. Cheng, N. Ghorbani, R. Fernandez, N. Nagappan, and P. C. Rigby, “Ai-assisted code author- ing at scale: Fine-tuning, deploying, and mixed methods evaluation,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 1066–1085, 2024

    2024

  7. [7]

    Amazon q developer: The most capable generative ai–powered assistant for software development,

    A. Q. Developer, “Amazon q developer: The most capable generative ai–powered assistant for software development,” 2024, accessed: 2025-05-20. [Online]. Available: https://aws.amazon.com/q/developer/

    work page 2024

  8. [8]

    Intellij ai editor,

    Jetbrains, “Intellij ai editor,” 2024, accessed: 2025-05-20. [Online]. Available: https://www.jetbrains.com/help/idea/use-ai-in-editor.html

    work page 2024

Show all 56 references
  1. [9]

    Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x,

    Q. Zheng, X. Xia, X. Zou, Y . Dong, S. Wang, Y . Xue, L. Shen, Z. Wang, A. Wang, Y . Liet al., “Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x,” inProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Min...

    2023

  2. [10]

    On the effectiveness of large language models in domain-specific code generation,

    X. Gu, M. Chen, Y . Lin, H. Zhang, C. Wan, Z. Wei, Y . Xu, and J. Wang, “On the effectiveness of large language models in domain-specific code generation,”ACM Trans. Softw. Eng. Methodol., vol. 34, no. 3, pp. 1–22, 2025, demonstrates task/domain variability among CodeLLMs acro...

    2025

  3. [11]

    Cruxeval-x: A benchmark for multilingual code reasoning, understanding and execution,

    R. Xu, J. Cao, Y . Lu, H. Lin, X. Han, B. He, S.-C. Cheung, and L. Sun, “Cruxeval-x: A benchmark for multilingual code reasoning, understanding and execution,”arXiv preprint arXiv:2408.13001, 2024

    2024 arXiv

  4. [12]

    Mceval: Massively multilingual code evaluation,

    L. Chai, S. Liu, J. Yang, Y . Yin, K. Jin, J. Liu, T. Sun, G. Zhang, C. Ren, H. Guoet al., “Mceval: Massively multilingual code evaluation,”arXiv preprint arXiv:2406.07436, 2024

    2024 arXiv

  5. [13]

    Mellum-4b-base,

    N. Pavlichenko, I. Nazarov, I. Dolgov, E. Garanina, K. Lasocki, J. Reshetnikova, S. Boitsov, I. Bondyrev, D. Karaeva, M. Sheptyakov, D. Ustalov, A. Mukhin, S. Proshev, N. Abramov, O. Kolomyttseva, K. Lysaniuk, I. Zavidnyi, A. Semenkin, V . Tankov, and U. Sazanovich, “Mellum-4b...

    2025

  6. [14]

    Full line code completion in jetbrains ides: All you need to know,

    JetBrains, “Full line code completion in jetbrains ides: All you need to know,” 2024, accessed: 2025-05-20. [Online]. Available: https://blog.jetbrains.com/blog/2024/04/04/ full-line-code-completion-in-jetbrains-ides-all-you-need-to-know/ #under-the-hood

    2024

  7. [15]

    On the effectiveness of large language models in domain- specific code generation,

    X. Gu, M. Chen, Y . Lin, Y . Hu, H. Zhang, C. Wan, Z. Wei, Y . Xu, and J. Wang, “On the effectiveness of large language models in domain- specific code generation,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 3, pp. 1–22, 2025

    2025

  8. [16]

    Domaineval: An auto-constructed benchmark for multi-domain code generation,

    Q. Zhu, J. Cao, Y . Lu, H. Lin, X. Han, L. Sun, and S.-C. Cheung, “Domaineval: An auto-constructed benchmark for multi-domain code generation,” inProceedings of the AAAI Conference on Artificial Intel- ligence, vol. 39, no. 24, 2025, pp. 26 148–26 156

    2025

  9. [17]

    Evocodebench: An evolving code generation benchmark with domain-specific evaluations,

    J. Li, G. Li, X. Zhang, Y . Zhao, Y . Dong, Z. Jin, B. Li, F. Huang, and Y . Li, “Evocodebench: An evolving code generation benchmark with domain-specific evaluations,”Advances in Neural Information Processing Systems, vol. 37, pp. 57 619–57 641, 2024

    2024

  10. [18]

    How well do llms generate code for different application domains? benchmark and evaluation,

    D. Zheng, Y . Wang, E. Shi, H. Zhang, and Z. Zheng, “How well do llms generate code for different application domains? benchmark and evaluation,”arXiv preprint arXiv:2412.18573, 2024

    2024 arXiv

  11. [19]

    Full line code completion: Bringing ai to desktop,

    A. Semenkin, V . Bibaev, Y . Sokolov, K. Krylov, A. Kalina, A. Khan- nanova, D. Savenkov, D. Rovdo, I. Davidenko, K. Karnaukhovet al., “Full line code completion: Bringing ai to desktop,”arXiv preprint arXiv:2405.08704, 2024

    2024 arXiv

  12. [20]

    Multi-line ai-assisted code authoring,

    O. Dunay, D. Cheng, A. Tait, P. Thakkar, P. C. Rigby, A. Chiu, I. Ahmad, A. Ganesan, C. Maddila, V . Muraliet al., “Multi-line ai-assisted code authoring,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 150– 160

    2024

  13. [21]

    An empirical analysis of the utilization of mul- tiple programming languages in open source projects,

    P. Mayer and A. Bauer, “An empirical analysis of the utilization of mul- tiple programming languages in open source projects,” inProceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, 2015, pp. 1–10

    2015

  14. [22]

    Multi-programming- language commits in oss: an empirical study on apache projects,

    Z. Li, X. Qi, Q. Yu, P. Liang, R. Mo, and C. Yang, “Multi-programming- language commits in oss: an empirical study on apache projects,” in2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). IEEE, 2021, pp. 219–229

    2021

  15. [23]

    A large scale study of multiple programming languages and code quality,

    P. S. Kochhar, D. Wijedasa, and D. Lo, “A large scale study of multiple programming languages and code quality,” in2016 IEEE 23Rd inter- national conference on software analysis, evolution, and reengineering (SANER), vol. 1. IEEE, 2016, pp. 563–573

    2016

  16. [24]

    Multi-language software development: Issues, challenges, and solutions,

    H. Yang, Y . Nong, S. Wang, and H. Cai, “Multi-language software development: Issues, challenges, and solutions,”IEEE Transactions on Software Engineering, vol. 50, no. 3, pp. 512–533, 2024

    2024

  17. [25]

    Multipl-e: A scalable and extensible approach to benchmarking neural code generation,

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y . Zi, C. J. Anderson, M. Q. Feldmanet al., “Multipl-e: A scalable and extensible approach to benchmarking neural code generation,”arXiv preprint arXiv:2208.08227, 2022

    2022 arXiv

  18. [26]

    Llms are helpful with python, but what about all of the other programming languages?

    T. Dunn, “Llms are helpful with python, but what about all of the other programming languages?” 2023, accessed: 2025-05-20. [Online]. Available: https://blog.continue.dev/programming-languages/

    2023

  19. [27]

    Eu ai act: first regulation on ar- tificial intelligence,

    E. Parliament, “Eu ai act: first regulation on ar- tificial intelligence,” 2023, accessed: 2025-05-20. [On- line]. Available: https://www.europarl.europa.eu/topics/en/article/ 20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence

    2023

  20. [28]

    Unveiling security, privacy, and ethical concerns of chatgpt,

    X. Wu, R. Duan, and J. Ni, “Unveiling security, privacy, and ethical concerns of chatgpt,”Journal of Information and Intelligence, vol. 2, no. 2, pp. 102–115, 2024

    2024

  21. [29]

    Security and privacy challenges of large language models: A survey,

    B. C. Das, M. H. Amini, and Y . Wu, “Security and privacy challenges of large language models: A survey,”ACM Computing Surveys, vol. 57, no. 6, pp. 1–39, 2025

    2025

  22. [30]

    Ray serve model multiplexing,

    Ray, “Ray serve model multiplexing,” 2025, accessed: 2025-05-20. [Online]. Available: https://docs.ray.io/en/latest/serve/ model-multiplexing.html

    2025

  23. [31]

    Handle concurrent requests in ollama,

    Ollama, “Handle concurrent requests in ollama,” 2025, accessed: 2025-05-20. [Online]. Available: https://github.com/ollama/ollama/blob/ main/docs/faq.md

    2025

  24. [32]

    Get started with kserve modelmesh for multi-model serving,

    IBM, “Get started with kserve modelmesh for multi-model serving,” 2023, accessed: 2025-05-

    2023

  25. [33]

    Available: https://developer.ibm.com/tutorials/ awb-get-started-with-kserve-modelmesh-for-multi-model-serving/

    [Online]. Available: https://developer.ibm.com/tutorials/ awb-get-started-with-kserve-modelmesh-for-multi-model-serving/

  26. [34]

    Faaswap: Slo-aware, gpu-efficient serverless inference via model swapping,

    M. Yu, A. Wang, D. Chen, H. Yu, X. Luo, Z. Li, W. Wang, R. Chen, D. Nie, and H. Yang, “Faaswap: Slo-aware, gpu-efficient serverless inference via model swapping,”arXiv preprint arXiv:2306.03622, 2024. [Online]. Available: https://arxiv.org/abs/2306.03622

    2024 arXiv

  27. [35]

    Cace replication package,

    CACE, “Cace replication package,” 2025, accessed: 2025-05-20. [Online]. Available: https://figshare.com/s/1a30559050bc43d5fefd

    2025

  28. [36]

    Closing the gap: A user study on the real- world usefulness of ai-powered vulnerability detection & repair in the ide,

    B. Steenhoek, K. Sivaraman, R. S. Gonzalez, Y . Mohylevskyy, R. Z. Moghaddam, and W. Le, “Closing the gap: A user study on the real- world usefulness of ai-powered vulnerability detection & repair in the ide,”arXiv preprint arXiv:2412.14306, 2024

    2024 arXiv

  29. [37]

    Future of programming with ai — lex fridman podcast 447,

    L. Fridman and C. Team, “Future of programming with ai — lex fridman podcast 447,” 2024, accessed: 2025-05-20. [Online]. Available: https://lexfridman.com/cursor-team-transcript

    2024

  30. [38]

    Multi-step jailbreaking privacy attacks on chatgpt,

    H. Li, D. Guo, W. Fan, M. Xu, J. Huang, F. Meng, and Y . Song, “Multi-step jailbreaking privacy attacks on chatgpt,”arXiv preprint arXiv:2304.05197, 2023

    2023 arXiv

  31. [39]

    Privacy and security challenges in large language models,

    V . Rathod, S. Nabavirazavi, S. Zad, and S. S. Iyengar, “Privacy and security challenges in large language models,” in2025 IEEE 15th Annual Computing and Communication Workshop and Conference (CCWC). IEEE, 2025, pp. 00 746–00 752

    2025

  32. [40]

    Ai privacy risks & mitigations large language models (llms),

    E. D. P. Board, “Ai privacy risks & mitigations large language models (llms),” 2023, accessed: 2025-05-20. [Online]. Available: https://www.edpb.europa.eu/our-work-tools/our-documents/ support-pool-experts-projects/ai-privacy-risks-mitigations-large en

    2023

  33. [41]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Luet al., “Qwen2. 5-coder technical report,”arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  34. [42]

    Codestral: Empowering developers and democratising coding with mistral ai,

    M. AI, “Codestral: Empowering developers and democratising coding with mistral ai,” May 2024, accessed: 2025-05-20. [Online]. Available: https://mistral.ai/news/codestral

    2024

  35. [43]

    The redmonk programming language rankings: June 2024,

    RedMonk, “The redmonk programming language rankings: June 2024,” 2024, accessed: 2025-05-20. [Online]. Available: https://redmonk.com/ sogrady/2024/09/12/language-rankings-6-24/

    2024

  36. [44]

    Tiobe index for may 2025,

    TIOBE, “Tiobe index for may 2025,” 2025, accessed: 2025-05-20. [Online]. Available: https://www.tiobe.com/tiobe-index/

    2025

  37. [45]

    Developers & ai coding assistant trends,

    CodeSignal, “Developers & ai coding assistant trends,” 2023, accessed: 2025-05-20. [Online]. Available: https://codesignal.com/ report-developers-and-ai-coding-assistant-trends/

    2023

  38. [46]

    The state of developer ecosystem 2023,

    JetBrains, “The state of developer ecosystem 2023,” 2023, accessed: 2025-05-20. [Online]. Available: https://www.jetbrains.com/ lp/devecosystem-2023/

    2023

  39. [47]

    Developer sentiment around ai/ml,

    StackOverflow, “Developer sentiment around ai/ml,” 2023, accessed: 2025-05-20. [Online]. Available: https://stackoverflow.co/ labs/developer-sentiment-ai-ml/

    2023

  40. [48]

    Exegpt: Constraint-aware resource scheduling for llm inference,

    H. Oh, K. Kim, J. Kim, S. Kim, J. Lee, D.-s. Chang, and J. Seo, “Exegpt: Constraint-aware resource scheduling for llm inference,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2024, pp...

    2024

  41. [49]

    {ServerlessLLM}:{Low-Latency}serverless inference for large language models,

    Y . Fu, L. Xue, Y . Huang, A.-O. Brabete, D. Ustiugov, Y . Patel, and L. Mai, “{ServerlessLLM}:{Low-Latency}serverless inference for large language models,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 135–153

    2024

  42. [50]

    Model hotswapping: Optimizing ai infrastructure and enhancing llm efficiency,

    Snowflake, “Model hotswapping: Optimizing ai infrastructure and enhancing llm efficiency,” 2025, accessed: 2025-05-20. [Online]. Available: https://www.snowflake.com/en/engineering-blog/ llm-interference-model-hotswapping/

    2025

  43. [51]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 611–626

    2023

  44. [52]

    Pie: Pooling cpu memory for llm inference,

    Y . Xu, Z. Mao, X. Mo, S. Liu, and I. Stoica, “Pie: Pooling cpu memory for llm inference,”arXiv preprint arXiv:2411.09317, 2024

    2024 arXiv

  45. [53]

    Flexgen: High-throughput generative inference of large language models with a single gpu,

    Y . Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. R ´e, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 31 094–31 116

    2023

  46. [54]

    Towards understanding the mixture-of-experts layer in deep learning,

    Z. Chen, Y . Deng, Y . Wu, Q. Gu, and Y . Li, “Towards understanding the mixture-of-experts layer in deep learning,”Advances in neural information processing systems, vol. 35, pp. 23 049–23 062, 2022

    2022

  47. [55]

    Fiddler: Cpu- gpu orchestration for fast inference of mixture-of-experts models,

    K. Kamahori, T. Tang, Y . Gu, K. Zhu, and B. Kasikci, “Fiddler: Cpu- gpu orchestration for fast inference of mixture-of-experts models,”arXiv preprint arXiv:2402.07033, 2024

    2024 arXiv

  48. [56]

    Deepspeed-moe: Advancing mixture- of-experts inference and training to power next-generation ai scale,

    S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y . Aminabadi, A. A. Awan, J. Rasley, and Y . He, “Deepspeed-moe: Advancing mixture- of-experts inference and training to power next-generation ai scale,” inInternational conference on machine learning. PMLR, 2022, pp. 18 332–18 346

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.