{"id":"092aef13-0caa-4923-8be5-8a3c91d6a943","arxiv_id":"2504.15303","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A scheduling and deployment optimization system for LLM inference on heterogeneous GPU/NPU clusters that reports 33.6% to 122.5% throughput gains over round-robin in two small testbeds.","lead":"This paper presents a system for serving large language models on clusters with different types of AI accelerators. It combines a profiling-based search for the best deployment layout with a scheduler that routes requests based on each instance's compute speed and memory pressure.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Deployment optimizer's order-preserving assumption is the weakest link: validated on one model/cluster, and Algorithm 1's printed return formula is internally inconsistent.","rationale":"The paper's headline is a combined claim, and the strongest_claim is fair: if the system works as described, a lightweight profiling approach plus a capacity-aware scheduler would be useful. The scheduler has some genuine evidence: actual vLLM runs, two clusters, and comparison against four baselines. The deployment half, however, is the distinctive contribution and is the place where the paper is least secure. The authors themselves flag that the estimator is not accurate and rely on an order-preserving property that is only demonstrated in one narrow configuration. My review adds two concrete observations that make this concern sharper rather than merely a request for more evidence. First, the text of Section 5.1 cannot be reconciled with Figure 4 as written: the estimated order contains a duplicate t=8 and is said to align with an actual order that contains t=4. Second, Algorithm 1's return statement is inconsistent with the derivation in Section 3.2; it uses the tensor-parallelism degree ti as the multiplier instead of the number of instances ui/ti. This is not a stylistic issue: it changes the estimated throughput by a factor of ti^2/ui and, if followed literally, would bias the search toward high tensor-parallelism configurations. Because the deployment optimizer is supposed to replace expensive benchmarks, its validation must be strong; currently it is a single-order check with internal inconsistencies and no error bars or code. The scheduler's WRR comparison is a secondary concern: at high request rates a hand-set weighted round-robin beats the proposed scheduler, so the headline percentages are rate-dependent. Still, the primary reason for a conditional verdict is the deployment-optimizer ranking guarantee, and the recommended test would settle whether that guarantee actually holds on more than one setup.","tokens_in":18360,"tokens_out":8451,"duration_ms":79164,"concrete_test":"Re-run the deployment comparison in Section 5.1 and the multi-machine experiment in Section 5.3 using the corrected system-throughput formula `(ui / ti) * tokenNum / totalTime` instead of line 17's `ti * tokenNum / totalTime`. Include at least one additional model, such as DeepSeek-R1-Distill-Qwen-14B or Llama-3-70B, and both clusters (8x V100 and V100+A800). For each feasible ti, compute the estimated ranking from Algorithm 1 and compare it with measured vLLM throughput under continuous batching, using the same ShareGPT sample plus one finite request rate such as 16 req/s, and at least 5 random seeds. The concern is resolved only if the corrected estimator's ranking matches the measured ranking for every configuration in both clusters; any inversion, especially between t=1 and t=2 or t=4, demonstrates that the exhaustive-search-without-benchmarks claim is not reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the central 'lightweight profiling replaces expensive benchmarks' claim is Section 3.2's assumption that the static-batching estimator preserves the ranking of vLLM continuous-batching configurations. The paper explicitly concedes the estimate is not accurate (Section 3.2) and offers only one validation (Section 5.1): one machine, one model, four tensor-parallelism choices, two random seeds, and a deliberately balanced request distribution. That is weak support for a general ranking property, and the section is internally inconsistent: the measured order is stated as t=2, t=4, t=1, t=8, while the text says the estimated order is 't = 2, t = 8, t = 1, and t = 8' (with t=8 repeated), which cannot align with the measured sequence. More concretely, Algorithm 1 as printed returns `ti · (tokenNum / totalTime)` on line 17, whereas the derivation in Section 3.2 requires `(ui / ti) · (tokenNum / totalTime)` for system throughput. If line 17 is what was run, Figure 4 does not validate the described estimator; if it is a typo, the paper should correct it and re-check. No multi-machine ranking evidence is given for the heterogeneous cluster in Section 5.3, so the deployment optimizer could select a wrong configuration there. The scheduler results are also rate-dependent, with WRR beating the proposed scheduler at request rates 24 and infinity in Section 5.2, but the deployment-optimizer issue is more fundamental because a ranking failure directly negates the claimed contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-part system for serving LLMs on heterogeneous clusters. The first component is an offline deployment-configuration optimizer that profiles per-instance prefill/decode latency models, estimates batch processing times under a static-batching assumption, and exhaustively searches tensor-parallelism degrees. The second component is a runtime request scheduler that uses the same per-instance latency models together with KV-cache utilization to compute request workloads and places each request so as to minimize the maximum instance load. Experiments on two small clusters report scheduler throughput improvements of 122.5% and 33.6% over round-robin scheduling, and a single-machine experiment is used to argue that the static-batching estimator preserves the configuration ranking of vLLM's continuous batching. The core claimed mechanism is that lightweight profiling can replace expensive throughput benchmarks for deployment search.","tokens_in":18710,"tokens_out":6300,"duration_ms":56078,"significance":"If the claimed ranking property and the scheduler gains hold, the work would be practically useful: it outlines a reproducible, lightweight profiling-based alternative to exhaustive benchmark-based configuration search and a workload-aware scheduler for mixed-accelerator clusters. The modeling is clearly specified and the system components are sensible, and the paper honestly concedes in Section 3.2 that the estimator is inaccurate and relies on an order-preserving property. I also credit the authors for designing a deliberately balanced test setup in Section 5.1 to isolate configuration throughput. However, the evidence is currently thin: results are self-reported without code, data, or error bars; the single-machine ranking validation has an internal inconsistency; and the heterogeneous-cluster experiment does not validate the deployment optimizer. These issues matter for the central claims and need to be addressed in revision.","major_comments":[{"comment":"The printed return statement in Algorithm 1, `return ti·(tokenNum/totalTime)`, is inconsistent with the system-throughput derivation in Section 3.2. The system has pi = ui/ti instances, so system throughput is (ui/ti)·(tokenNum/totalTime), not ti·(tokenNum/totalTime). In the single-machine experiment (ui=8) this changes the ranking criterion by a factor that depends on ti, so Figure 4 as documented does not validate the estimator described in the text. Please correct Algorithm 1 and re-run the ranking validation, or explicitly clarify the notation if the printed formula is not what was executed.","section":"Section 3.2, Algorithm 1"},{"comment":"The load-bearing assumption for the deployment optimizer is the order-preserving property between the static-batching estimator and vLLM's continuous batching; the paper itself states \"the estimation is not accurate\" in Section 3.2. The evidence in Section 5.1 is limited to one machine, one model, four tensor-parallelism choices, two random seeds, and two request samples. Moreover, the text reports the two estimated rankings as both being \"t = 2, t = 8, t = 1, and t = 8\", which is not a valid permutation and cannot be consistent with the measured order \"t = 2, t = 4, t = 1, t = 8\". As printed, the only validation of the ranking assumption is unusable. Please fix the sequence, add more diverse validation (for example, a heterogeneous multi-machine case where the chosen configuration is compared against alternatives), and report the actual estimated and measured throughput values.","section":"Section 3.2 and Section 5.1"},{"comment":"The headline improvements (122.5% and 33.6%) are single request-rate points, not aggregate results. In Section 5.2 the weighted round-robin baseline is reported as the highest-throughput strategy at request rates 24 and infinity, while the proposed scheduler is best only at rates 8 and 16; the 122.5% figure is quoted for rate 24 alone. The figures appear to report single runs without error bars or multiple seeds, so the claimed gains may not be statistically robust. Please report multiple trials with confidence intervals, show the full request-rate sweep, and state in the abstract and conclusion which rates the headline gains refer to.","section":"Section 5.2 and Section 5.3"},{"comment":"Section 5.3 evaluates the scheduler on a two-machine heterogeneous cluster with a fixed deployment (four t=2 V100 instances and one t=1 A800 instance), but it does not compare this deployment against alternative tensor-parallelism configurations or demonstrate that the deployment optimizer selected it. Since deployment configuration optimization is the first contribution claimed in the introduction, the end-to-end system claim lacks its central validation. The authors should either run the configuration optimizer on the heterogeneous cluster and compare predicted versus measured rankings, or explicitly present Section 5.3 as a scheduler-only evaluation and temper the corresponding contribution claims.","section":"Section 5.3"}],"minor_comments":[{"comment":"The symbols `wchoice_r` and `wc_r` are undefined; they should be a single notation such as `w^c_r`, the workload of request r on the chosen instance c.","section":"Algorithm 2, lines 15 and 17"},{"comment":"Equation (5) omits the per-parameter byte size `bbyte` that appears in Equation (1); as written, KV Size(r) has units of hidden units rather than bytes, which matters for the FP16 deployment described in Section 5.1.","section":"Equation (5)"},{"comment":"The predictor uses `numpy.random.normal` to draw output lengths from the fitted mean and standard deviation; this can in principle produce negative output lengths, so the authors should either clamp the predictions or report that no negative values occurred in the experiments.","section":"Section 5.2, output-length predictor"},{"comment":"The phrase \"Ref [19]\" is an unusual citation format and should be written as a standard citation, e.g., \"[19]\".","section":"Section 4, output-length predictor discussion"}],"recommendation":"major_revision","confidential_remarks":"The direction is publishable, but the current manuscript's experimental validation is too thin for the central claims and contains internal inconsistencies (Algorithm 1 and the Section 5.1 ranking statement). I would not reject if the authors can correct these errors and supply stronger validation, but as presented the evidence is not sufficient to support the abstract's claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the scheduler workload rule — per-request time scaled by an exponential KV-cache-pressure penalty — is a sensible new heuristic, and the single-machine two-instance experiment suggests it balances load well. The deployment optimizer is more fragile: its order-preserving assumption is tested on one model and one cluster, with two seeds, and the paper contains an internal inconsistency in the algorithm as printed.\n\nCredit where due: the authors state plainly that the static-batching estimator is not accurate (Sec. 3.2) and that their case for it rests on ranking, not absolute values. That is the right way to frame a cheap profiling-based config search. The workload formula in Eq. (7) is new to me; the memory-pressure term is a reasonable mechanism for steering requests away from instances whose KV cache is filling. The experiments, though small, show the failure mode of round-robin clearly.\n\nThe soft spots are real. No code, no data, no error bars; the testbeds are two machines (8x V100, and 4x V100 + 1x A800). The ranking validation in Sec. 5.1 uses one model, four TP choices, and two random seeds — weak support for a general order-preserving claim, and no ranking evidence on the heterogeneous cluster. Worse, the text garbles the estimated order ('t = 2, t = 8, t = 1, and t = 8') and Algorithm 1 as printed returns ti·(tokenNum/totalTime) where Sec. 3.2's derivation needs (ui/ti)·(tokenNum/totalTime). If line 17 is what was run, Fig. 4 is not validating the described estimator; if it is a typo, it needs fixing and the evaluation may need to be redone. The scheduler results are also rate-dependent: the stronger baseline (WRR) wins at rates 24 and infinity in Sec. 5.2, which is fine — they acknowledge it — but it undercuts the headline '122.5%' as a single-point claim rather than a demonstrated property.\n\nWho is this for: anyone building a serving layer for mixed accelerator pools. The paper is worth a serious referee. It needs major revision: a corrected algorithm, a clearly reproduced ranking test (ideally on the heterogeneous cluster), and some form of artifact release. I would not desk-reject it, but I would not accept it near its current form.","headline":"A sensible scheduler heuristic and a fragile deployment optimizer; the paper deserves a serious referee but needs a corrected algorithm and stronger ranking evidence.","tokens_in":19215,"tokens_out":2200,"would_cite":false,"duration_ms":19648,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight profiling model plus a KV-cache-aware request scheduler can lift LLM inference throughput on heterogeneous clusters by 122.5% and 33.6% over round-robin scheduling.","keywords":["large language model inference","heterogeneous clusters","request scheduling","deployment configuration optimization","KV cache","tensor parallelism","continuous batching","throughput"],"falsifier":"Run the deployment search on a three-machine cluster with three different accelerator types and compare the estimated ranking of tensor-parallel configurations against the measured throughput of each configuration on the same inference engine; if the estimated top configuration is not the measured best, or any adjacent pair is inverted, the core deployment claim fails.","tokens_in":1463,"feed_emoji":"⚡","tokens_out":1497,"duration_ms":68187,"temperature":0.7,"pith_summary":"The paper tries to establish that heterogeneous LLM clusters can be made fast without expensive benchmark suites or hand-tuned routing weights: two lightweight mechanisms suffice. First, a deployment optimizer models memory and batch-processing time from fitted linear formulas, exhaustively searches tensor-parallel settings, and claims that the ranking of configurations is preserved even though the absolute estimates are inaccurate. Second, a runtime scheduler computes a per-request workload from processing time weighted by KV-cache occupancy and greedily assigns each request to the instance that minimizes the maximum load. The paper reports throughput gains of 122.5% and 33.6% over round-robin on two-machine heterogeneous clusters. If correct, this means existing mixed-accelerator infrastructure can be used closer to its limits with only profiling data and no per-hardware tuning expertise.","feed_headline":"Scheduler lifts LLM throughput by 122% on mixed GPU clusters","feed_subtitle":"Profiling-based deployment search plus KV-aware routing beats round-robin on uneven accelerators.","key_machinery":"The load-bearing objects are the linear time model and the workload formula. The linear model says that, for a fixed batch, prefill time is linear in batch size and total input length while each decode iteration is linear in batch size and cached length, giving constants $p^s_1$ through $p^s_8$ that are fitted from a few profiled runs. That model supplies the deployment estimator and the scheduler's per-request processing time $T^s_r$. The workload formula $w^s_r = T^s_r \\cdot e^{\\theta \\cdot kv_{usage}(s)}$ fuses that capacity estimate with current memory pressure: the exponential KV-cache term grows as the instance fills up, so the scheduler favors strong compute under moderate load and shifts toward memory-aware routing under bursts, while the mapper greedily assigns each request to minimize the largest resulting instance workload.","core_discovery":"The central claim is that both hard parts of serving on a heterogeneous cluster—choosing how to split each machine's accelerators into instances, and routing requests once those instances are live—can be solved with profiling data instead of exhaustive benchmarking or static weights. For deployment, the paper fits eight constants per machine to a linear model of prefill and decode time, checks memory feasibility, and evaluates every candidate tensor-parallel degree on a few sampled requests; the chosen degree matches the actual best in experiments even though the estimated throughput is far off. For scheduling, it defines the workload of request $r$ on instance $s$ as $T^s_r \\cdot e^{\\theta \\cdot kv_{usage}(s)}$, where $T^s_r$ is the per-request processing time under an ideal batch and $kv_{usage}(s)$ is the fraction of KV cache occupied, then sends each request to the instance that keeps the maximum resulting load smallest. Experiments on two-machine clusters with different GPU types report a 122.5% throughput gain over round-robin in the two-instance setting and a 33.6% gain in the multi-machine setting.","pith_inferences":["A natural extension is to test whether the order-preserving property of the deployment estimator holds beyond the two random seeds tested; if continuous batching is monotone in instance throughput, the ranking argument could be proven rather than demonstrated.","The output-length predictor used in experiments is deliberately simple, sampling from the dataset's distribution; coupling the scheduler with a learned predictor would be a direct testable upgrade and could determine whether the reported gains transfer to workloads with skewed output lengths.","The fixed penalty parameter $\\theta = 2$ is a tuning choice; sweeping $\\theta$ per cluster and request rate would show whether the scheduler is robust or needs per-deployment calibration.","The paper treats deployment optimization and request scheduling sequentially, but the size of the reported routing gain suggests the two should be optimized jointly, since configuration choices change instance capacities and therefore the best routing policy."],"forward_implications":["A cluster operator can select the best tensor-parallel degree per machine without running full throughput benchmarks, using only a small profiling pass and an exhaustive search over configurations.","The scheduler can balance unequal instances without manually assigned weights, which the paper notes are hard to choose when a cluster contains multiple GPU models.","Round-robin routing degrades sharply as instance capacities diverge, so heterogeneous clusters gain disproportionately from capacity-aware scheduling.","The workload formula adapts automatically to request bursts: at low occupancy it routes by computing power, and as KV caches fill it routes by available memory.","The measured gains of 122.5% and 33.6% are concrete improvements over round-robin on the tested clusters, and the paper's comparison shows that naive round-robin can even lag behind sending every request to the stronger instance."],"supporting_citations":[{"why":"Supplies the linear prefill and decode time model used both for deployment throughput estimation and for the scheduler's per-request workload calculation.","marker":"[5]"},{"why":"Provides the least-squares curve fitting used to obtain the eight per-machine constants in the time model.","marker":"[13]"},{"why":"Introduces continuous batching, the engine behavior that makes the static-batching estimate inaccurate while motivating why order-preserving ranking is the meaningful property.","marker":"[17]"},{"why":"Underlies the engine's memory management and KV-cache mechanisms that the deployment memory constraint and the schedule's KV-usage term depend on.","marker":"[8]"},{"why":"Lists existing serving stacks that use round-robin request distribution, the baseline that the proposed scheduler is compared against.","marker":"[1,2,3]"}],"fun_headline_variants":["KV-aware routing lifts LLM throughput 122% on mixed GPU clusters","Profiling beats benchmarks for LLM serving on heterogeneous GPUs","Heterogeneous cluster LLM serving: smarter config and scheduling","122% LLM throughput gain via profiling and KV-aware scheduling","LLM inference speedup on uneven clusters with profiling-based search"],"cache_read_input_tokens":21376,"weakest_assumption_plain":"The deployment optimizer's usefulness rests on the claim that its static-batching throughput estimate ranks configurations in the same order as the real engine's continuous batching, but the paper admits the estimate itself is inaccurate and checks the ordering in only one setup with two random seeds.","fun_headline_variants_meta":{"raw":{"variants":["KV-aware routing lifts LLM throughput 122% on mixed GPU clusters","Profiling beats benchmarks for LLM serving on heterogeneous GPUs","Heterogeneous cluster LLM serving: smarter config and scheduling","122% LLM throughput gain via profiling and KV-aware scheduling","LLM inference speedup on uneven clusters with profiling-based search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1506,"prompt_tokens":979,"completion_tokens":527,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":439}},"tokens_in":595,"tokens_out":527,"duration_ms":4899,"temperature":1.0,"reasoning_tokens":439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:04:45.242840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the deployment search on a three-machine cluster with three different accelerator types and compare the estimated ranking of tensor-parallel configurations against the measured throughput of each configuration on the same inference engine; if the estimated top configuration is not the measured best, or any adjacent pair is inverted, the core deployment claim fails.","supporting_citations":[{"cited_title":"arXiv preprint arXiv:2406.1351 1 (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the linear prefill and decode time model used both for deployment throughput estimation and for the scheduler's per-request workload calculation."},{"cited_title":"Nature methods 17(3), 261–272 (2020)","cited_arxiv_id":null,"evidence_quote":"Provides the least-squares curve fitting used to obtain the eight per-machine constants in the time model."},{"cited_title":"In: 16th USENIX Symposium on Oper- ating Systems Design and Implementation (OSDI 22)","cited_arxiv_id":null,"evidence_quote":"Introduces continuous batching, the engine behavior that makes the static-batching estimate inaccurate while motivating why order-preserving ranking is the meaningful property."}],"review_version":1}