LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusters

Mario Colosi; Reza Farahani; Schahram Dustdar; Zoha Azimi

REVIEW 5 major objections 4 minor 14 references

LLM queries on heterogeneous edge clusters are best served by choosing model, quantization, and device per query, minimizing predicted response time under accuracy and resource constraints.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:49 UTC pith:LNVX7VTQ

load-bearing objection Solid systems paper with a useful benchmark and a clear direction, but the evaluation leaks training data into the live workload and the accuracy predictor's device dependence undermines the QoS argument. the 5 major comments →

arxiv 2607.17175 v1 pith:LNVX7VTQ submitted 2026-07-19 cs.DC

LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusters

Reza Farahani , Zoha Azimi , Mario Colosi , Schahram Dustdar This is my paper

classification cs.DC

keywords LLM inferenceedge orchestrationQoS-aware schedulingmodel quantizationbinary integer linear programmingML-based predictionlatency minimizationresource-constrained devices

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LMEdge claims that serving LLM queries on a heterogeneous edge cluster should be orchestrated per query: choose the model family, size, quantization level, and execution device together, using predicted latency, accuracy, and resource usage. The paper formalizes this as a binary integer linear program that minimizes predicted response time under accuracy and resource constraints, then approximates it with a greedy heuristic that runs in sub-second time. On a 57-instance container-orchestrated testbed with 29 model configurations and over 1,400 queries, the paper reports lower response times, stable accuracy, higher resource utilization, and a serving ratio approaching that of a random baseline that ignores constraints. The reason to care: if this holds, modest heterogeneous edge devices can serve LLM queries with QoS guarantees without a large centralized GPU fleet.

Core claim

LMEdge's central claim is that per-query orchestration - choosing model family, size, quantization, and device together - beats routing that only balances load or ignores QoS. The objective is predicted response time T = F + ((δ_in + L)/μ)·(1+λ·l_i/ρ_i), with F predicted inference time, μ bandwidth, l_i active inferences, ρ_i concurrency bound. Assignments must keep predicted accuracy within (1−θ) of the best available and satisfy compute, memory, bandwidth, and concurrency limits. A greedy heuristic sorts queries by their best-case T and assigns the feasible triple with minimal T. On 57 instances the paper reports lower response time, preserved accuracy, better resource utilization, and hig

What carries the argument

The load-bearing mechanism is the predicted response-time identity of Eq. (2): for every query-model-quantization-device combination, response time is estimated as inference time plus a transmission term (input size plus predicted output size over bandwidth) inflated by a congestion factor (1+λ·active_inferences/concurrency_limit). Five lightweight ML predictors supply the inference time, accuracy, compute demand, memory demand, and output size for each combination; the BILP (Eq. 9) minimizes the sum of these predicted times subject to accuracy (Eq. 4) and resource constraints (Eqs. 5-8), and Algorithm 1 approximates it greedily in O(|Q|·|M|·|C|·|I|) time. The identity sets the scheduler's p

Load-bearing premise

The premise is that the five predictors - especially the inference-time model, reported with R²=0.71 - remain accurate during live scheduling, so that minimizing predicted response time really minimizes measured response time.

What would settle it

Hold out a strict subset of the 1,422 queries (or better, entire datasets) before training the five predictors, then run LMEdge's scheduler on that held-out set under the same Poisson traffic and bandwidth replay; if the measured response times of chosen assignments are not ordered by the predicted T, or if chosen assignments are no faster than the average feasible assignment, the optimization objective has not carried over to live operation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Edge clusters can serve LLM queries with per-query QoS (latency and accuracy) instead of coarse model-level or load-only routing.
Lightweight ML predictors are sufficient to drive online scheduling: millisecond prediction and sub-second heuristic runtimes make the approach practical under Poisson arrivals.
Resource-constrained devices (small boards and low-power nodes) can be used productively: the scheduler assigns simple queries to them and reserves larger models for accuracy-sensitive tasks.
Deferring queries that cannot meet accuracy or resource bounds is a deliberate QoS mechanism, and the paper shows the serving ratio remains close to unconstrained random routing at higher arrival rates.
The 59,000-row benchmark of query-model-quantization-device measurements is a reusable asset for comparing future edge LLM orchestration strategies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Extension: the latency gains depend on a hand-tuned congestion multiplier (1+λ·l_i/ρ_i) with λ=0.5 standing in for a queueing model; replacing it with measured queueing delay would test whether the gains persist.
Extension: because the accuracy predictor uses query and model features but not device features, the paper implicitly assumes device choice does not change correctness beyond the tolerated θ; a test on larger or newer hardware would check that assumption.
Extension: the 59,000 training rows come from the same 1,422 queries used in the live workload; holding out entire datasets would reveal whether scheduling gains survive distribution shift.
Extension: the architecture's registry and trace collection could support online retraining, so the five predictors would track query drift rather than relying on the static benchmark distribution.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid systems paper with a useful benchmark and a clear direction, but the evaluation leaks training data into the live workload and the accuracy predictor's device dependence undermines the QoS argument.

read the letter

Short version: this is a competent systems paper that deserves referee time, but the evaluation as written doesn't support the online claim as strongly as the abstract suggests. The new piece is the joint per-query choice over model family, size, quantization, and device, plus a 59K-row benchmark generated from 1,422 queries on a real 57-instance Kubernetes testbed. That benchmark, if released, could become a standard comparison point.

The formal machinery is fine. The BILP in Eqs. (1)–(9) is consistent, the greedy heuristic is clearly specified, and the testbed description is unusually concrete — Kubernetes versions, netem traces, device counts, model/quantization table. That level of detail is rare and useful.

Where it gets wobbly is evaluation. Section 6 uses the same 1,422 queries to build the 59K training records and to run the live workload, and Section 7 reports predictor R²/F1 without any described train/validation/test split. If those numbers are in-sample, the R²=0.71 for inference time is not evidence about online behavior. The paper needs a held-out split and, ideally, per-query or per-replicate error bars; Fig. 4 has single curves for Poisson arrivals.

The accuracy predictor is a conceptual problem. Eq. (3) defines A^{m,c}_{q,i} — accuracy as a function of the execution device. Semantically, correctness of a given model+quantization on a query should not depend on which machine runs it. If the predictor consumes device features (the formulation says it does), then the accuracy filter in line 20 of Alg. 1 can mis-rank assignments based on deployment artifacts. The stress test claims Fig. 3b shows device features are among the strongest accuracy predictors; reading the actual text, that's not what the paper says — linguistic/structural features dominate and model features have smaller impact. So that particular charge is wrong. But the design smell is real and worth an ablation.

Also, the abstract claims LMEdge increases serving ratio, but Fig. 4b shows Random serves the most queries; the paper itself admits LMEdge defers queries. That's an overstatement. The serving-ratio claim should be qualified.

Minor: the congestion multiplier (1+λ·l_i/ρ_i) is a hand-tuned patch, not a validated queuing model; fine as a heuristic, but should be described as such. And the exclusion of RouteLLM/OptLLM/C2MAB-V as "not directly comparable" is weakly argued — they're per-query routers, and the paper should justify why they can't run on this testbed.

Bottom line: the direction is plausible and the artifact is valuable, but the evaluation lacks the controls to prove the headline claim. A serious referee should ask for a held-out split, error bars, an optimality-gap check against BILP on small instances, and release of the dataset/code. I'd send it to review, not desk reject.

Referee Report

5 major / 4 minor

Summary. The paper presents LMEdge, an edge orchestration service that assigns each LLM query to a model family/size, quantization level, and execution device. The assignment problem is formulated as a BILP (Eqs. 1–9) minimizing predicted response time under accuracy, bandwidth, compute, memory, and concurrency constraints. For scalability, the authors train five ML predictors (inference time, accuracy, CPU/GPU usage, memory usage, response size) and propose a greedy heuristic (Alg. 1) that ranks queries and selects the cheapest feasible assignment. The system is evaluated on a Kubernetes testbed with 57 instances, 29 model configurations, and 1,422 queries from five datasets, generating a 59,000-row benchmark. The authors report reduced latency, preserved accuracy, improved resource utilization, and higher serving ratio relative to Random and Load-aware baselines, plus much lower scheduling time than the exact BILP.

Significance. If the results are sound, LMEdge is a useful contribution: it tackles the realistic problem of joint model-quantization-device selection in heterogeneous edge clusters, provides a reproducible 59,000-row dataset, and gives an unusually concrete testbed description (Kubernetes 1.32.3, Ollama, netem 4G traces, exact node counts). The BILP formulation is internally consistent, and the heuristic is a standard, clearly specified greedy algorithm. However, the central evaluation claims are weakened by methodological gaps in the predictor evaluation and by an apparent inconsistency between the reported serving-ratio result and the abstract. These issues need to be resolved before the core claims can be accepted.

major comments (5)

[§6, §7] The paper does not describe any train/validation/test split for the five ML predictors. Section 6 states that the evaluation workload consists of 1,422 queries from five datasets, and that the 59,000-row benchmarking dataset is built from 'query, model, quantization, and device combinations' using those datasets. Section 7 then reports predictor R² / F1 values (e.g., inference-time R²=0.71, accuracy F1=0.81) with no indication of whether these are in-sample or out-of-sample. If the 59,000 rows are generated from the same 1,422 queries used in the live scheduling experiment, the predictor metrics in Table 2 may be in-sample, and the latency/accuracy gains reported in Fig. 4 would not generalize to unseen queries. Please report a proper split (e.g., by query ID or time) and evaluate the online scheduler on a held-out query set.
[§3, Eq. (4); §7, Fig. 3b] The accuracy predictor includes device features ('Dev.' in Fig. 3b), yet Eq. (3) and Eq. (4) treat accuracy as a property of the query, model, and quantization (with superscript i). If execution device does not affect correctness, then device features in the accuracy model are fitting deployment artifacts and the accuracy constraint can mis-rank assignments arbitrarily as device features vary. This directly affects the correctness filter in Alg. 1, line 20, and hence the accuracy/latency trade-off measured in Fig. 4c. Please justify why device features belong in an accuracy predictor, or ablate them and show the accuracy model and scheduling results with and without device features.
[Abstract; §7, Fig. 4b] The abstract claims LMEdge 'increases serving ratio compared to two baselines,' but Fig. 4b shows that Random attains the highest serving ratio at all arrival rates, and the text admits that LMEdge 'deliberately defers queries that violate accuracy or congestion requirements.' The claim as stated is therefore contradicted by the paper's own results. Please qualify the serving-ratio claim (e.g., 'among QoS-aware approaches' or 'approaches Random's throughput at higher rates') and discuss the latency/accuracy versus serving-ratio trade-off explicitly.
[§3, Eq. (2); §6] The congestion-aware delay term (1 + λ·l_i/ρ_i) with a single hand-set λ=0.5 is used as a substitute for a queuing model, and it directly determines the objective minimized by both the BILP and Alg. 1. The paper provides no evidence that this simple multiplier produces response-time estimates that track measured latency under the Poisson workload and netem traces. Please report a calibration check (e.g., predicted T vs. measured end-to-end latency) and a sensitivity analysis over λ, or justify the choice with data.
[§7, Fig. 4] The performance comparisons report point estimates without confidence intervals, error bars, or statistical tests. Given the high variance typical of LLM inference on heterogeneous edge devices (visible in the broad inference-time distribution in Fig. 2d), claims such as 'LMEdge consistently achieves the lowest response times' (Fig. 4a) need quantitative support. Please report per-trial distributions, significance tests (e.g., bootstrap or paired tests), or at least standard errors.

minor comments (4)

[§7, Tables 2–3] There are formatting errors: 'Tables 2–2' should be 'Tables 2–3', and 'T able 3' has a stray space. Also, the tables do not report the train/validation/test split or cross-validation procedure for the predictor metrics.
[§6] The TruthfulQA binarization threshold of 0.7 is arbitrary and should be justified or varied in a sensitivity analysis, since it directly defines the accuracy labels used for both training and evaluation.
[§7] The BILP-vs-heuristic comparison reports 'speedups of up to 1565%' and '7012%' without stating the BILP's optimality gap or the time scale. Please clarify whether CPLEX solved the instances to proven optimality and report both solver time and heuristic time in absolute units.
[§7, Fig. 5] The heatmaps are informative but lack a colorbar and quantitative values, making it hard to assess the claimed differences in query distribution across model configurations. Please add a color scale and, if possible, a numeric matrix.

Circularity Check

0 steps flagged

No construction-level circularity: the reported latency/accuracy results are measured on a live testbed, not read off the fitted predictors; the shared query pool is a validation concern, not a derivation identity.

full rationale

The paper's derivation chain is: define a BILP objective (Eq. 9) in terms of predicted response time T (Eq. 2), approximate it with the greedy Alg. 1, and evaluate the resulting assignments against Random and Load-aware baselines. The key question is whether the evaluation outcomes are forced by the construction. They are not. The response-time, accuracy, resource-utilization, and serving-ratio results in Fig. 4 are measured outcomes of actually executing the selected queries on the Kubernetes testbed under replayed 4G traces, not the predicted values used inside the scheduler. A poor predictor would produce poor assignments and worse measured latency; the empirical comparison is therefore external to the fitted models. The accuracy constraint (Eq. 4) constrains predicted accuracy only, while the claimed accuracy preservation in Fig. 4c is a measured property. There is no load-bearing self-citation: the related work and method citations are to external prior systems, and no self-cited uniqueness theorem or ansatz is invoked to force the model choice. The congestion multiplier (1 + λ·l_i/ρ_i) in Eq. 2 is presented as a hand-tuned control term, not imported from the authors' prior work. The main validity concern is that the 1,422 live-workload queries also generated the 59,000-row training data for the predictors, and Section 7 reports predictor metrics without a described train/validation/test split. This threatens generalization and the informativeness of the reported R² and F1 numbers, but it does not make the measured testbed comparison equal to the fitted inputs by construction. Under the required standard of exhibiting a specific reduction of a claimed result to its own inputs, no such reduction is present.

Axiom & Free-Parameter Ledger

8 free parameters · 8 axioms · 0 invented entities

The central claims rest on: five ML predictors trained on the 59K benchmark, the ad-hoc congestion model in Eq. (2) with hand-set λ=0.5, the accuracy-tolerance parameter θ=0.1, and a series of domain assumptions about the testbed (Poisson arrivals, replayed 4G traces, Ollama/Kubernetes behavior) and about the ground-truth datasets. No new physical entities are introduced; the scheduling service is a software artifact.

free parameters (8)

λ (congestion penalty) = 0.5
Hand-set multiplier in Eq. (2)'s congestion term (1+λ·l_i/ρ_i); shapes every predicted response time that the BILP minimizes and the heuristic sorts by. No sensitivity analysis reported.
θ (accuracy tolerance) = 0.1
Binds the accuracy constraint Eq. (4): assignments must reach at least (1−θ)·a_best. Directly controls the accuracy/serving-ratio trade-off; no sensitivity analysis.
TruthfulQA binarization threshold = 0.7
Semantic-similarity scores above 0.7 are labeled 'correct' to build the accuracy predictor's training targets; the choice changes the accuracy labels for one of the five datasets.
XGBoost hyperparameters (per task) = ntrees 100–200, d_max 5–7, ψ 0.1–0.3
Selected per predictor (Tables 2–3); these are fitted to the benchmark and determine the R²/F1 values that feed scheduling decisions.
ρ_i (per-device concurrency limit) = CPU cores or empirically determined max GPU streams
Used in the Eq. (2) denominator and constraint Eq. (8); empirically chosen per instance, affecting deferral and utilization numbers.
Scheduling epoch = 10 s
Fixed epoch length over which queries are batched; interacts with the Poisson arrival rates and the deferral mechanism in the measured latency.
Query feature vector φ_q = structural + linguistic features
Hand-designed features (length, sentence count, lexical diversity, unique words); no ablation links these features to end-to-end QoS outcomes.
59K-row sampling scheme = unspecified
Which of the feasible query-model-quantization-device triples were measured into the benchmark is not stated; coverage gaps would bias the predictors.

axioms (8)

domain assumption Eq. (2)'s multiplicative congestion term (1+λ·l_i/ρ_i) adequately captures queuing, serialization, and contention under load
Appears in the objective (Eq. 9) and the heuristic ranking (Alg. 1); it is an ad-hoc model, not a queuing-theoretic derivation, and λ=0.5 is hand-tuned.
domain assumption Accuracy of a (query, model, quantization) triple is predictable and device-dependent
The accuracy predictor consumes device features (Fig. 3b), yet correctness of an LLM output is independent of the host device; including device features risks learning spurious associations.
domain assumption Ground-truth labels of MMLU subsets, GSM8K, CommonsenseQA, TruthfulQA, and HumanEval are reliable accuracy measures
Used to label all 59K benchmark rows and to compute Fig. 4c; TruthfulQA correctness is defined by the 0.7 binarization.
domain assumption Capacity constraints (Eqs. 5–8) with non-preemptive execution capture real instance limits
The formulation omits context-window/token limits, KV-cache memory, and Ollama's internal batching behavior; assumed irrelevant to the comparison.
domain assumption Replayed 4G LTE traces via tc/netem reproduce realistic edge network dynamics
Bandwidth conditions are synthesized from reference [11]; the realism of the congestion/delay measurements depends on this replay.
domain assumption Poisson arrivals at 0.4–1.0 queries/s/instance represent realistic workloads
All three schedulers are compared under this synthetic traffic; no real trace workload is used.
standard math BILP is NP-hard; greedy assignment after ranking is an acceptable approximation framework
Alg. 1 is a standard greedy; the paper relies on this without quantifying the optimality gap.
domain assumption 10-second Prometheus/cAdvisor monitoring granularity supports QoS decisions
Resource states used in constraints are up to 10 s stale at scheduling time; staleness is not analyzed.

pith-pipeline@v1.3.0-alltime-deepseek · 9378 in / 25726 out tokens · 220870 ms · 2026-08-01T18:49:17.038789+00:00 · methodology

0 comments

read the original abstract

Large language model (LLM) services increasingly operate on edge infrastructure, enabling low-latency and privacy-preserving AI services. However, efficiently serving LLM requests across heterogeneous and resource-constrained edge devices require orchestration mechanisms that jointly determine model configuration (family, size, and quantization level) and execution placement while satisfying user- and system-level quality of service (QoS) requirements. This paper introduces LMEdge, a QoS-aware orchestration service that dynamically makes these decisions across heterogeneous edge devices. We formulate the problem as a binary integer linear programming (BILP) optimization that minimizes response time under accuracy, network, and resource constraints. To enable scalable online scheduling, we employ five lightweight machine learning (ML) models to predict query-specific latency, accuracy, resource usage, and response size for each model-size-quantization-device combination, and design a lightweight heuristic that approximates the BILP solution. We collect a comprehensive benchmarking dataset of over 59000 rows to train models and support reproducibility. Evaluation on a Kubernetes-based edge testbed with 57 instances and diverse query categories shows that LMEdge reduces latency, preserves accuracy, improves resource utilization, and increases serving ratio compared to two baselines.

Figures

Figures reproduced from arXiv: 2607.17175 by Mario Colosi, Reza Farahani, Schahram Dustdar, Zoha Azimi.

**Figure 2.** Figure 2: Distribution of query complexity, accuracy, and inference time in the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Figure 3: Feature importance across the five prediction models. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

**Figure 4.** Figure 4: Performance of LMEdge vs. baselines under different query arrival rates. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

**Figure 5.** Figure 5: Query distribution across model configurations in different approaches. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

14 extracted references · 3 linked inside Pith

[2]

Demandsage,https://www.demandsage.com/chatgpt-statistics/, Retrieved: 2026-05-29

2026
[3]

NemoCurator Prompt Task and Complexity Classifier.https://huggingface.co/ nvidia/prompt-task-and-complexity-classifier, accessed: 2026-05-29

2026
[4]

In: Intl

Cai, F., et al.: Edge-llm: A Collaborative Framework for Large Language Model Serving in Edge Computing. In: Intl. Conf. on Web Services. IEEE (2024)

2024
[5]

arXiv preprint arXiv:2405.16587 (2024)

Dai, X., et al.: Cost-Effective Online Multi-LLM Selection with Versatile Reward Models. arXiv preprint arXiv:2405.16587 (2024)

Pith/arXiv arXiv 2024
[6]

arXiv preprint arXiv:2505.16508 (2025)

Jang, S., Morabito, R.: Edge-First Language Model Inference: Models, Metrics, and Tradeoffs. arXiv preprint arXiv:2505.16508 (2025)

Pith/arXiv arXiv 2025
[7]

In: Intl

Liu, Y., et al.: OptLLM: Optimal Assignment of Queries to Large Language Mod- els. In: Intl. Conf. on Web Services. IEEE (2024)

2024
[8]

In: Proc

Oh, H., et al.: ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference. In: Proc. of the 29th ACM Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (2024)

2024
[9]

In: The 13th Intl

Ong, I., et al.: RouteLLM: Learning to Route LLMs from Preference Data. In: The 13th Intl. Conf. on Learning Representations (2024)

2024
[10]

In: 51st Intl

Patel, P., et al.: Splitwise: Efficient Generative LLM Inference using Phase Split- ting. In: 51st Intl. Symp. on Computer Architecture. IEEE (2024)

2024
[11]

In: Proc

Raca,D.,etal.:BeyondThroughput:A4GLTEDatasetwithChannelandContext Metrics. In: Proc. of the 9th ACM Multimedia Systems Conf. (2018)

2018
[12]

In: Intl

Stojkovic, J., et al.: DynamoLLM: Designing LLM Inference Clusters for Perfor- mance and Energy Efficiency. In: Intl. Symp. on High Performance Computer Ar- chitecture. IEEE (2025)

2025
[13]

arXiv preprint arXiv:2501.14205 (2025)

Xu, M., et al.: Serving Long-Context LLMs at the Mobile Edge: Test-Time Rein- forcement Learning-based Model Caching and Inference Offloading. arXiv preprint arXiv:2501.14205 (2025)

Pith/arXiv arXiv 2025
[14]

Yao, Z., et al.: Enhancing LLM QoS through Cloud-Edge Collaboration: A Diffusion-based Multi-Agent Reinforcement Learning Approach. Tran. on Services Computing (2025)

2025
[15]

IEEE Internet of Things Journal (2024)

Zhang, M., et al.: Edgeshard: Efficient LLM Inference via Collaborative Edge Com- puting. IEEE Internet of Things Journal (2024)

2024

[1] [2]

Demandsage,https://www.demandsage.com/chatgpt-statistics/, Retrieved: 2026-05-29

2026

[2] [3]

NemoCurator Prompt Task and Complexity Classifier.https://huggingface.co/ nvidia/prompt-task-and-complexity-classifier, accessed: 2026-05-29

2026

[3] [4]

In: Intl

Cai, F., et al.: Edge-llm: A Collaborative Framework for Large Language Model Serving in Edge Computing. In: Intl. Conf. on Web Services. IEEE (2024)

2024

[4] [5]

arXiv preprint arXiv:2405.16587 (2024)

Dai, X., et al.: Cost-Effective Online Multi-LLM Selection with Versatile Reward Models. arXiv preprint arXiv:2405.16587 (2024)

Pith/arXiv arXiv 2024

[5] [6]

arXiv preprint arXiv:2505.16508 (2025)

Jang, S., Morabito, R.: Edge-First Language Model Inference: Models, Metrics, and Tradeoffs. arXiv preprint arXiv:2505.16508 (2025)

Pith/arXiv arXiv 2025

[6] [7]

In: Intl

Liu, Y., et al.: OptLLM: Optimal Assignment of Queries to Large Language Mod- els. In: Intl. Conf. on Web Services. IEEE (2024)

2024

[7] [8]

In: Proc

Oh, H., et al.: ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference. In: Proc. of the 29th ACM Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (2024)

2024

[8] [9]

In: The 13th Intl

Ong, I., et al.: RouteLLM: Learning to Route LLMs from Preference Data. In: The 13th Intl. Conf. on Learning Representations (2024)

2024

[9] [10]

In: 51st Intl

Patel, P., et al.: Splitwise: Efficient Generative LLM Inference using Phase Split- ting. In: 51st Intl. Symp. on Computer Architecture. IEEE (2024)

2024

[10] [11]

In: Proc

Raca,D.,etal.:BeyondThroughput:A4GLTEDatasetwithChannelandContext Metrics. In: Proc. of the 9th ACM Multimedia Systems Conf. (2018)

2018

[11] [12]

In: Intl

Stojkovic, J., et al.: DynamoLLM: Designing LLM Inference Clusters for Perfor- mance and Energy Efficiency. In: Intl. Symp. on High Performance Computer Ar- chitecture. IEEE (2025)

2025

[12] [13]

arXiv preprint arXiv:2501.14205 (2025)

Xu, M., et al.: Serving Long-Context LLMs at the Mobile Edge: Test-Time Rein- forcement Learning-based Model Caching and Inference Offloading. arXiv preprint arXiv:2501.14205 (2025)

Pith/arXiv arXiv 2025

[13] [14]

Yao, Z., et al.: Enhancing LLM QoS through Cloud-Edge Collaboration: A Diffusion-based Multi-Agent Reinforcement Learning Approach. Tran. on Services Computing (2025)

2025

[14] [15]

IEEE Internet of Things Journal (2024)

Zhang, M., et al.: Edgeshard: Efficient LLM Inference via Collaborative Edge Com- puting. IEEE Internet of Things Journal (2024)

2024