{"id":"6cc5feb2-e9ec-4991-bc72-601157b46200","arxiv_id":"2506.05871","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"A dispatch-aware roofline simulator picks the best LLM serving strategy by predicted goodput without running costly GPU benchmarks.","lead":"BestServe is a lightweight estimator that ranks LLM serving strategies by predicted goodput, covering both collocated and disaggregated setups in minutes on a CPU. It maps decode-phase bottlenecks to CPU-GPU dispatch instead of memory, but its headline 20% error claim is contradicted by its own 30.1% result on long-generation workloads.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper validates per-strategy goodput errors but never checks whether BestServe's top-ranked strategy matches the ground-truth optimal strategy, so the central claim is unverified; OP4's 30.1% error and in-sample parameter fitting make this gap load-bearing.","rationale":"The reader correctly identifies the pseudo batch size heuristic (Eq. 9) as the likely cause of OP4's 30.1% error. My stress-test focuses one level higher: even if that heuristic were replaced by a token-level simulation, the paper would still not establish the central claim because it never validates the decision output (which strategy is selected) and because the accuracy numbers come from a calibration/evaluation loop on the same data. A framework that estimates goodput with 20% average error can still pick the right strategy if errors are rank-preserving; conversely, a 30.1% error in one scenario can be tolerable for top-1 selection if the ranking is unchanged. The paper provides no evidence either way, so the contribution's headline promise is unverified rather than disproven. This does not invalidate the simulation approach; the paper is candid about limitations and the modular design is plausible. Code and data are withheld, which increases the importance of a top-1 agreement check that the authors could run internally. Overall, the correct verdict remains CONDITIONAL: the method may work, but the validation should be tightened before the 20% and optimal-strategy claims are accepted.","tokens_in":24210,"tokens_out":6283,"duration_ms":63800,"concrete_test":"Using the data behind Figure 11, compute for each scenario OP1-OP4 the serving strategy with maximum predicted normalized goodput and the strategy with maximum ground-truth normalized goodput; if the predicted argmax matches the ground-truth argmax in all four scenarios, additionally refit ec, em, and e+ on OP1-OP3 and evaluate on OP4. If top-1 agreement fails, or if the out-of-sample OP4 error remains above 20%, the abstract's claim that BestServe determines the optimal strategy within a 20% error margin is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"BestServe's purpose is strategy selection: the Optimizer (Algorithms 8-9) enumerates configurations and returns the one with maximum simulated goodput. Section 4.3 reports only average absolute relative errors of predicted goodput per strategy (11.2%, 12.1%, 8.6%, 30.1% for OP1-OP4) and never reports whether the argmax strategy under BestServe equals the argmax in the manual benchmarking ground truth. A 20% (or even 30.1%) average error does not establish ranking correctness, and Section 5 states that the pseudo batch size heuristic (Eq. (9), tau=2.5) overestimates goodput especially for collocation strategies in OP4, exactly the setting where a collocation-versus-disaggregation recommendation could invert. Compounding this, the efficiency parameters ec, em, e+ used for the reported predictions were aligned on the same manual benchmarking data that serves as ground truth (Section 4.1, Hyperparameters), so the four scenarios are not an independent test of the claimed 20% margin. The central claim is therefore under-supported even if Eq. (9) were fixed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"BestServe proposes a three-level framework (Estimator, Simulator, Optimizer) for ranking LLM serving strategies in collocated and disaggregated architectures by predicted goodput. The Estimator uses an adapted roofline model with MFU/MBU efficiency parameters and explicitly models CPU-GPU dispatch latency; the Simulator approximates request-level dynamics of prefill and decode phases with a pseudo batch size heuristic; and the Optimizer enumerates configurations and uses bisection on the arrival rate to find the maximum goodput satisfying SLO constraints. The paper validates on vLLM-Ascend with the CodeLlama-34b-Instruct model on Ascend 910B3 NPUs across four fixed-length operating scenarios, reporting average absolute relative errors of 11.2%, 12.1%, 8.6%, and 30.1% for OP1-OP4. The stated goal is to eliminate costly case-by-case benchmarking by determining the optimal serving strategy in minutes on a standard CPU.","tokens_in":24529,"tokens_out":5907,"duration_ms":59857,"significance":"If the framework's ranking accuracy were established, BestServe would be a practically valuable tool for early-stage deployment planning, as it replaces expensive trial-and-error benchmarking with lightweight simulation. The paper's strengths include its modular hierarchical design, explicit modeling of dispatch-bound decode behavior, and a detailed, mostly self-contained presentation of the estimator's operator-level cost tables in the appendices. The paper is also candid in Section 5 about the decode-phase heuristic and parameter-tuning limitations. However, the central claim is currently under-supported: the validation measures per-strategy goodput error rather than strategy-ranking accuracy; the efficiency parameters are calibrated on the same manual benchmarking data used as ground truth; and the abstract's 20% error claim is contradicted by the OP4 result. These issues are load-bearing for the paper's main contribution and must be addressed before the framework's usefulness for strategy selection can be accepted.","major_comments":[{"comment":"The paper's stated purpose is strategy selection, but the validation reports only the per-strategy average absolute relative error of predicted goodput. It never reports whether the strategy with maximal predicted goodput matches the strategy with maximal ground-truth goodput in each operating scenario, nor any rank correlation between predicted and ground-truth orderings. A per-strategy mean error of 11.2-30.1% does not by itself guarantee correct ranking: strategies with similar goodput can be swapped without a large mean error, and Section 5 states that the OP4 error is concentrated in collocated strategies. Please report, for each scenario, the ground-truth and predicted argmax strategies and the goodput gap between them, and provide a rank correlation (e.g., Spearman) over the enumerated strategies. Also state the uncertainty in the manually interpolated ground truth, which the paper's own Figure 10a suggests can fluctuate by about 5% in P90 TTFT.","section":"§4.3 and Figure 11"},{"comment":"The abstract's claim of 'achieving predictions within a 20% error margin' is not supported by the reported numbers: §4.3 gives average absolute relative errors of 11.2%, 12.1%, 8.6%, and 30.1% for OP1-OP4, so OP4 violates the stated margin by a wide margin. The paper itself attributes this to the pseudo batch size heuristic in §5. The abstract and the conclusion should be revised to a qualified claim (e.g., 'within 20% for scenarios with short-to-moderate generation lengths'), or the validation must be extended to show that decode-heavy scenarios are also within the claimed margin.","section":"Abstract and §4.3"},{"comment":"The efficiency parameters ec, em, and e+ are determined in §4.1 by aligning the simulator's intermediate results (Figures 6 and 8) with real inference data from manual benchmarking, and the same manual benchmarking data are then used as the ground truth in §4.3. The reported error rates are therefore a measure of in-sample fit rather than predictive accuracy. Please state this explicitly, quantify how sensitive the predicted goodput and the selected strategy are to perturbations of these parameters within plausible ranges, and, if feasible, hold out one or more operating scenarios or strategies during calibration and report errors on the held-out cases.","section":"§4.1 vs §4.3"},{"comment":"The bisection loop condition in Algorithm 8, 'while λu − λℓ < ε do', is inverted; it should be 'while λu − λℓ > ε do' so that the loop terminates when the interval is sufficiently small. As written, the pseudocode enters the loop when the difference is already below the tolerance and would not terminate. Please correct the algorithm and confirm that the implemented Optimizer uses the correct stopping condition.","section":"§3.5, Algorithm 8"},{"comment":"The pseudo batch size heuristic with the single global scalar τ=2.5 is the current bottleneck for decode-heavy scenarios. Section 5 states that in OP4 this heuristic overestimates goodput 'especially for serving strategies using the collocation architecture,' which is precisely the regime where a collocation-versus-disaggregation recommendation could invert. Please add a sensitivity analysis over τ (for example, τ ∈ {1.5, 2.5, 4.0}) and over the feasibility relaxation factor in Algorithm 9, showing whether the top-ranked strategy changes, and report OP4 ranking quality separately for collocated and disaggregated strategies.","section":"Eq. (9) and §5"}],"minor_comments":[{"comment":"The name 'BestServe' is frequently concatenated with surrounding words (e.g., 'WepresentBestServe', 'BestServeleverages', 'BestServedemonstrates'); please fix the LaTeX spacing.","section":"Throughout"},{"comment":"The sentence 'linear regressions are conducted exploit the linear relationship' should read 'conducted to exploit the linear relationship'.","section":"§4.1"},{"comment":"Section 3.4.2 states that τ=2.5 provides a reasonable approximation, but §4.1's hyperparameter list mentions only MFU, MBU, and communication efficiency; please clarify how τ was set in the validation and whether it was tuned per scenario.","section":"§3.4.2 and §4.1"},{"comment":"The sentence 'We still use (12) to estimate the computation time of the prefill phase of Attention module when TP is considered' should refer to the decode phase, since Eq. (12) is the decode-phase formula.","section":"Appendix B.2"},{"comment":"The histograms are sorted by predicted goodput but do not visually distinguish collocated from disaggregated strategies; adding markers or labels for the two architecture types would make it easier to see where ranking errors occur.","section":"Figure 11"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a systems/ML venue. The central concern is validation: the missing ranking-level correctness check, the in-sample calibration of efficiency parameters, and the OP4 violation of the abstract's 20% claim together leave the main contribution unverified. These issues are fixable with additional analysis of the existing data and a more careful statement of claims, so I recommend major revision rather than rejection. Note also that the availability statement prevents code/data release; given that, the pseudocode error in Algorithm 8 should be treated seriously by the authors in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BestServe has a real idea: relabeling the decode phase as dispatch-bound rather than memory-bound, and building an adapted roofline model with CPU-to-device dispatch dynamics into a tandem-queue simulator. That framing is the most interesting part, and the per-operator work/memory tables give it a concrete foundation. The paper is also unusually candid; Section 5 admits the pseudo-batch heuristic is why OP4 misses by 30.1%.\n\nWhat it does well: the three-level design (estimator, simulator, optimizer) is clean, the collocated simulator's prefill-priority and suspend/resume logic is more detailed than DistServe's analytic model, and the runtime claim (minutes on a CPU vs manual HPC benchmarking) is credible for a ~3k-line Python simulator. Credit where due: the authors report four scenarios, give per-scenario errors, and explicitly flag the main limitation.\n\nThe soft spots are real and load-bearing. First, the abstract promises \"within a 20% error margin,\" but the validation reports 30.1% average error for OP4, so the headline is overstated as written. Second, the efficiency parameters (ec, em, e+) are tuned on the same manual-benchmark data that serves as ground truth. That makes the four scenarios an in-sample check, not an independent test. Third—and this is the stress-test point that holds up—the paper never checks whether the strategy BestServe ranks first actually matches the ground-truth optimal strategy. Per-strategy absolute error does not establish ranking correctness, and ranking is the entire purpose of the optimizer. OP4 is exactly the case where the pseudo-batch heuristic overestimates collocated goodput, so the framework could recommend the wrong architecture. The missing code/data (Huawei policy) makes it harder to audit, though the pseudocode is detailed enough to reimplement.\n\nNone of this kills the central idea. The dispatch-bound relabel and simulator structure are worth building on. But the accuracy claim and the ranking claim are under-supported as written. A serious referee should ask for: honest per-scenario error reporting, an argmax or rank-correlation check against ground truth, validation on held-out scenarios or a calibration split, and code or a step-by-step calibration guide.\n\nWho gets value: ML serving researchers and deployment planners who want a fast first-pass estimator before committing GPUs. It deserves peer review, with the above as required revisions.","headline":"A useful dispatch-aware goodput simulator with an overstated headline and a key missing check: does its top-ranked strategy match ground truth?","tokens_in":25080,"tokens_out":1910,"would_cite":false,"duration_ms":21804,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BestServe claims that the optimal LLM serving strategy — architecture, instance ratio, and parallelism — can be determined in minutes on a standard CPU by simulating inference with an adapted roofline model, replacing costly…","keywords":["LLM serving","goodput","prefill-decode disaggregation","roofline model","serving simulator","continuous batching","dispatch-bound decode","SLO-aware optimization"],"falsifier":"Take a long-generation, short-input workload like the paper's OP4, run the simulator and the real serving system over several prefill-to-decode ratios and collocated configurations, and compare which strategy achieves the highest measured goodput. If the simulated rank order disagrees with the measured rank order — not just the magnitude of the error — the pseudo batch heuristic is misleading for the very decisions the tool is built to make. A more direct test isolates the decode phase: measure per-request token-generation latency as the number of concurrently served requests grows, and check whether the ratios follow the $b^\\dagger$ curve for generation lengths of 1024 or more.","tokens_in":24009,"feed_emoji":"⚙️","tokens_out":13330,"duration_ms":108670,"temperature":0.7,"pith_summary":"BestServe claims that choosing how to serve a large language model — which architecture, how many prefill versus decode instances, and what tensor-parallel size — can be automated by simulation instead of trial-and-error benchmarking. The paper builds a three-level analyzer: an estimator that predicts operator-level latencies from an adapted roofline model, a simulator that tracks requests through prefill and decode stages under SLO constraints, and an optimizer that binary-searches the highest sustainable arrival rate. The central assertion is that the optimal serving strategy can be found in minutes on a single standard CPU, with measured average absolute errors of 11.2%, 12.1%, 8.6%, and 30.1% across four operating scenarios. The motivation is that neither architecture wins universally; the better choice depends on request patterns such as input length, generation length, and arrival rate.","feed_headline":"Simulation, not benchmarks, picks the best LLM serving setup","feed_subtitle":"No GPU benchmarking needed: goodput predictions run in minutes on a standard CPU.","key_machinery":"The load-bearing object is the hierarchical analyzer: an Estimator built on an adapted roofline model, $P = \\min\\{e_c S_c, I e_m S_m\\}$, which says an operator runs at the smaller of its compute peak and its memory-bandwidth peak, each discounted by an efficiency factor; a Simulator that treats disaggregated serving as a tandem queue and reproduces prefill-prioritizing, non-mixing scheduling for collocated serving; and an Optimizer that bisects arrival rates to find goodput. The two mechanisms that make the whole thing fast are the pseudo batch size $b^\\dagger = \\max(\\lfloor(b+1)/\\tau\\rfloor, 1)$ with $\\tau=2.5$, which approximates continuous batching interference without token-level loops, and caching of the latency oracle by its functional arguments.","core_discovery":"The paper's central claim is that the maximum goodput of an LLM serving strategy — the highest request arrival rate whose P90 time-to-first-token and time-per-output-token stay within SLO bounds — can be estimated by lightweight simulation instead of measured by repeated benchmark runs. BestServe decomposes inference into operator-level work and memory traffic, applies an adapted roofline model with hardware efficiency parameters (MFU and MBU), and explicitly charges the time for the CPU to dispatch operators to the GPU, which the paper argues makes the decode phase dispatch-bound rather than memory-bound. A pseudo batch size heuristic, $b^\\dagger = \\max(\\lfloor(b+1)/\\tau\\rfloor, 1)$ with $\\tau = 2.5$, stands in for the full dynamics of continuous batching so each request is simulated once rather than token by token. An optimizer then bisects on arrival rate to produce the goodput of every candidate strategy. Validation against manual benchmarking across four operating scenarios reports average absolute relative errors of 11.2%, 12.1%, 8.6%, and 30.1%, with the largest error in the long-generation scenario attributed to the decode heuristic.","pith_inferences":["The paper's own data fixes the failure regime: expect errors near 30% whenever generation length is long and input short (its OP4), and much lower errors in the opposite corner where prefill dominates. The abstract's 20% margin is a claim about typical scenarios, not a uniform bound.","The single global value $\\tau=2.5$ in the pseudo batch size is the most promising lever: a generation-length-dependent or occupancy-dependent $\\tau$ could recover OP4-style accuracy while keeping one simulation loop per request.","Because BestServe ignores memory capacity entirely, its 'optimal' strategy should be read as optimal up to memory feasibility; strategies it ranks first could still fail to load on small-memory hardware, especially under PagedAttention-style dynamic caching.","The dispatch-bound account yields a concrete prediction the authors only gesture at: reducing CPU-to-GPU dispatch overhead (for example by fusing operators) should improve decode-phase goodput roughly in proportion to the dispatch share of estimated latency, and BestServe's estimator could quantify that gain before any code is written."],"forward_implications":["Service providers can compare collocation versus disaggregation, and different prefill-to-decode instance ratios, for their own traffic mix without renting GPUs for trial-and-error benchmarking.","Strategy search becomes a minutes-scale CPU job, so the optimal configuration can be re-derived as request patterns, model sizes, or SLOs change.","The framework backs the finding that disaggregation is not uniformly better than collocation: the winner depends on input length, generation length, and SLO thresholds, and BestServe predicts which configuration wins in each scenario.","If the dispatch-bound characterization of decode is correct, decode-phase hardware and software tuning should target CPU-GPU dispatch overhead rather than raw memory bandwidth alone."],"supporting_citations":[{"why":"Provides the roofline model that the estimator adapts with efficiency parameters to predict operator latency.","marker":"[33]"},{"why":"Introduces the goodput metric and the disaggregated prefill-decode design that BestServe evaluates against.","marker":"[36]"},{"why":"The collocated serving engine whose prefill-prioritizing, non-mixing scheduler the collocation simulator mimics.","marker":"[18]"},{"why":"The serving platform used to profile ground-truth goodput values that predictions are validated against.","marker":"[31]"},{"why":"The reference implementation for disaggregated serving that defines the comparison baseline.","marker":"[30]"},{"why":"Introduces iteration-level (cellular) batching, the continuous-batching behavior the pseudo batch size approximates.","marker":"[11]"},{"why":"The model family whose operator work and memory-traffic tables the estimator is built on.","marker":"[21]"},{"why":"Supplies tensor parallelism, the parallelism scheme whose communication costs the estimator models.","marker":"[28]"},{"why":"The queueing-theory source for the initial upper bound on arrival rate used by the optimizer's bisection.","marker":"[13]"}],"fun_headline_variants":["Simulation, not benchmarks, finds best LLM serving strategy","BestServe sim predicts LLM goodput in minutes on CPU","No GPU benchmarking: sim picks optimal LLM serving setup","Lightweight CPU sim estimates LLM serving goodput within 20%","BestServe: goodput optimization via simulation, not trial-and-error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework stands on one assumption: that a single heuristic formula, the pseudo batch size $b^\\dagger = \\max(\\lfloor(b+1)/\\tau\\rfloor, 1)$ with $\\tau = 2.5$, correctly captures how much slower token generation gets when requests share a GPU, for every generation length and load level.","fun_headline_variants_meta":{"raw":{"variants":["Simulation, not benchmarks, finds best LLM serving strategy","BestServe sim predicts LLM goodput in minutes on CPU","No GPU benchmarking: sim picks optimal LLM serving setup","Lightweight CPU sim estimates LLM serving goodput within 20%","BestServe: goodput optimization via simulation, not trial-and-error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1599,"prompt_tokens":896,"completion_tokens":703,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":629}},"tokens_in":512,"tokens_out":703,"duration_ms":6470,"temperature":1.0,"reasoning_tokens":629,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:12:38.748755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a long-generation, short-input workload like the paper's OP4, run the simulator and the real serving system over several prefill-to-decode ratios and collocated configurations, and compare which strategy achieves the highest measured goodput. If the simulated rank order disagrees with the measured rank order — not just the magnitude of the error — the pseudo batch heuristic is misleading for the very decisions the tool is built to make. A more direct test isolates the decode phase: measure per-request token-generation latency as the number of concurrently served requests grows, and check whether the ratios follow the $b^\\dagger$ curve for generation lengths of 1024 or more.","supporting_citations":[{"cited_title":"Roofline: an insightful visual performance model for multicore architectures.Commun","cited_arxiv_id":null,"evidence_quote":"Provides the roofline model that the estimator adapts with efficiency parameters to predict operator latency."},{"cited_title":"DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving","cited_arxiv_id":null,"evidence_quote":"Introduces the goodput metric and the disaggregated prefill-decode design that BestServe evaluates against."},{"cited_title":"Efficient memory management for large language model serving with PagedAttention","cited_arxiv_id":null,"evidence_quote":"The collocated serving engine whose prefill-prioritizing, non-mixing scheduler the collocation simulator mimics."},{"cited_title":"https://github.com/vllm-project/vllm-ascend","cited_arxiv_id":null,"evidence_quote":"The serving platform used to profile ground-truth goodput values that predictions are validated against."},{"cited_title":"https://docs.vllm.ai/en/v0.4.2/index.html","cited_arxiv_id":null,"evidence_quote":"The reference implementation for disaggregated serving that defines the comparison baseline."},{"cited_title":"Low latency rnn inference with cellular batching","cited_arxiv_id":null,"evidence_quote":"Introduces iteration-level (cellular) batching, the continuous-batching behavior the pseudo batch size approximates."},{"cited_title":"Llama 3.2: Revolutionizing edge AI and vision with open, customizable models","cited_arxiv_id":null,"evidence_quote":"The model family whose operator work and memory-traffic tables the estimator is built on."},{"cited_title":"Shortle, James M","cited_arxiv_id":null,"evidence_quote":"The queueing-theory source for the initial upper bound on arrival rate used by the optimizer's bisection."}],"review_version":1}