{"id":"8d109562-d2d3-4fef-9caa-44edaff68be0","arxiv_id":"2608.13057","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Expert cost in MoE serving follows a two-regime max-affine law, and a makespan-aware dispatcher over this model tracks the best fixed policy and wins where regimes mix.","lead":"This paper measures expert cost in Mixture-of-Experts serving on two GPU generations and shows it is flat in tokens up to roughly 160 tokens per expert, then linear, so balancing tokens alone or experts alone is wrong in the other regime. The authors build TEMPO, a makespan-aware dispatcher with a phase diagram that predicts when adaptive dispatch helps, and confirm the prediction on two flagship models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 92–100% mixed-regime claim is incompatible with the stated n*≈156 at B=128: no expert can exceed 128 tokens, so 0% of tokens can be linear; the claim must rely on a different, unidentifiable calibration.","rationale":"I read the paper in good faith and find the core engineering contributions credible: the wall-clock microbenchmark, the honest Section 7 limitation list, the like-for-like LPLB port, and the repeated replication checks all support the claim that TEMPO tracks the best fixed policy in the tested regimes and beats proxies in the flat regime. The reader's weakest_assumption—model transfer from the 8-GPU Testbed A calibration to EP32–64 and the full serving pipeline—is a real concern, and the reader correctly notes that the largest wins are model-scored extrapolations. My stress-test sharpens this into a more specific, internal problem: the paper's motivating fact that both regimes coexist in 92–100% of real decode batches is not computable with the headline n*≈156 at the reported batch sizes. Under standard MoE routing, an expert can receive at most one token per sequence, so at B=128 tokens/GPU, n_e ≤ 128 < n*, forcing 0% of tokens into the linear regime. The only way to reproduce the reported statistics is to use the full-pipeline fit, whose n*≈46 is derived from parameters that Section 5.2 shows are not individually identifiable (corr(â,b̂)=−0.997). The paper therefore relies on two different inflections—one measured for the isolated kernel and one unidentifiable for the deployed pipeline—without stating which one underlies Fig. 5. This does not by itself falsify the dispatch algorithm, which has independent wall-clock support in the flat regime, but it does mean the central narrative and the phase-diagram win region are not anchored to a single measured hardware boundary. The appropriate disposition is conditional acceptance: the authors should disclose which n* produced Fig. 5, provide the trace-level n_e distributions, and re-base the mixed-regime and phase-diagram claims on identifiable parameters. If the 92–100% claim depends on the collinear full-pipeline fit, the abstract and strongest_claim as stated overreach.","tokens_in":27952,"tokens_out":22591,"duration_ms":246057,"concrete_test":"Recompute the mixed-regime fractions of Fig. 5 from the recorded per-expert token counts n_e, using (a) the offline kernel inflection n* = b/β ≈ 156 from Table 6 and (b) the full-pipeline inflection n* = b/β from Table 7. If version (a) gives 0% mixed at B=128—which is forced by n_e ≤ 128—then the Section 2.4 '92–100%' sentence is false under the paper's stated n*. Also report a profile-likelihood confidence interval for the full-pipeline n*; if the interval spans roughly 14–52, the mixed-regime frequency is not a stable statistic. Finally, re-plot the phase-diagram win region using the identifiable quantities (the flat cost a+bG and β) instead of n*, and verify whether the 15.5% mixed-zone gains survive when the classifier is the one actually identified by the deployment fit.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central narrative rests on a pivotal empirical fact: at decode batch sizes, both regimes coexist in 92–100% of batches, so no fixed proxy is safe (Section 2.4, Fig. 5). This is mathematically impossible under the paper's own headline inflection. In decode, each active sequence contributes one token, and an expert can receive at most one token per sequence. At B=128 tokens/GPU, every expert therefore has n_e ≤ 128 tokens. The stated n* is ≈156–168 (Table 6, Fig. 2), so no expert can be in the linear regime: 0% of tokens are linear, not 20%+. The same conclusion holds if B counts routed token-expert incidences rather than unique tokens, since then unique tokens per GPU are even fewer. Consequently, the 92–100% mixed-batch statistic in Fig. 5 cannot have been computed with the headline n*. The only way to obtain the reported 'linear-region experts carry 91% of tokens' at B=512 is a much smaller per-expert inflection. The full-pipeline fit in Table 7 indeed implies n* = b/β = 16.59/0.358 ≈ 46 for DSv3, but Section 5.2 reports corr(â, b̂) = −0.997 with b ∈ [5.1, 18.8] (95% CI), so the full-pipeline n* is not identifiable and could range roughly 14–52. The paper thus conflates the measured kernel inflection (n*≈156) with a deployment-time inflection that is not a measured, identifiable quantity. Because the phase-diagram 'mixed zone' is where the up-to-15.5% extrapolated wins are claimed, this weakens the central motivation and the quantitative headline. The wall-clock flat-regime wins (B=32–128, Fig. 7) are not invalidated, but the paper's core framing—that realistic decode batches simultaneously straddle the hardware inflection—is not supported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies per-batch load balancing for expert-parallel (EP) Mixture-of-Experts serving, where the per-layer time is the makespan across GPUs. Measurements on two GPU generations show that per-expert fp8 grouped-GEMM time is flat below roughly 156-168 tokens (HBM weight streaming) and linear above, leading to a max-affine per-GPU cost model t_g = max(a+bG_g, c+beta N_g). The authors argue that real decode batches contain both regimes simultaneously, that no fixed token-count or expert-count proxy is therefore safe, and that the dispatch problem is a fixed-charge makespan problem that is NP-hard with two GPUs yet polynomial in each degenerate limit. They present TEMPO, a solver that runs in milliseconds off the critical path, integrates with SGLang via a fused in-graph kernel, and is validated by an 8-GPU microbenchmark, calibrated simulation, and end-to-end serving on Testbed B with Qwen3-235B (inside the predicted win region) and DeepSeek-V3 (outside). The paper's central claim is a phase diagram predicting when time-aware dispatch pays, rather than a universal win.","tokens_in":1625,"tokens_out":1842,"duration_ms":121195,"significance":"If the two-regime cost model is correct, the paper makes a substantive contribution: it explains why token-based and activation-based balancing fail in complementary regimes, provides a principled makespan objective, and delivers a deployable dispatcher with off-critical-path solving and a fused kernel. The paper is unusually careful about evidence: it includes an 8-GPU wall-clock microbenchmark with 93% pairwise ranking agreement, explicit negative controls, a like-for-like architectural decomposition of the LPLB gap, and honest limitations (L1-L6) distinguishing model-scored results from measured serving results. The NP-hardness and additive-approximation proofs are self-contained and appear sound. The main weakness is that the 'pivotal empirical fact' of 92-100% mixed-regime batches at decode sizes is not consistent with the stated kernel inflection, and the deployment-time inflection used for that statistic is not identifiable from the full-pipeline fit. This weakens the motivational narrative and the precision of the phase-diagram boundary, though the algorithmic and engineering contributions stand.","major_comments":[{"comment":"The claim that 92-100% of batches at B=128 tokens per GPU contain both regimes is mathematically incompatible with the stated inflection n* approximately 156-168. In decode, each active sequence contributes one token and an expert can receive at most one token per sequence, so with B=128 tokens per GPU every expert has n_e <= 128 < n*, meaning 0% of tokens can be in the linear regime. Yet Fig. 5(a) reports 92-100% mixed batches at B=128. The paper must specify which inflection was used as the classifier. If it is the full-pipeline value b/beta approximately 46 from Table 7, then Section 5.2 shows this quantity is not identifiable (corr(a_hat,b_hat) = -0.997, b in [5.1,18.8] microseconds at 95% CI), so the statistic is not robust. This is load-bearing because the paper's motivation states that the pivotal empirical fact is that regimes coexist in 92-100% of decode batches, and the phase-diagram win region is where regimes mix. Please correct the statistic, the stated n*, or the narrative, and report the sensitivity of the mixed-batch fraction to the choice of n*.","section":"Section 2.4, Fig. 5"},{"comment":"The paper conflates the measured kernel inflection (n* approximately 156) with the deployment-time inflection (b/beta approximately 46 for DSv3 from Table 7). The full-pipeline fit observes only G in [9,20], where a and b are nearly collinear, so b/beta could range roughly 14-52 within the reported 95% CI. This unidentifiability matters because the phase diagram (Fig. 6) and the analytic boundary B* of Section 3.2 depend on n*, and the scale extrapolation (Fig. 12) uses one point on the (a,b) ridge. The authors should demonstrate that the predicted win region and the B* boundary are stable when (a,b) are moved along the ridge within the CI, or state explicitly which calibration is used for each figure and why the phase-diagram conclusions are insensitive to the ridge.","section":"Section 5.2, Table 7, Section 3.2"},{"comment":"The abstract states that TEMPO stays within 1% of the best fixed baseline everywhere, but the wall-clock microbenchmark in Section 5.1 reports that TEMPO is within 5% (approximately run noise) of the per-B best fixed policy. The 1% figure is a model-space guarantee from Proposition 1 and the phase-grid evaluation, not a measured wall-clock result. Please qualify the abstract claim so that model-scored and wall-clock claims are not conflated.","section":"Abstract and Section 5.1"}],"minor_comments":[{"comment":"The caption should state explicitly which calibrated n* is used as the classifier and the source of the parameters, given that different sections use different n* values.","section":"Fig. 5 caption"},{"comment":"The analytic boundary formulas B*_avg and B*_hot should define all symbols (e.g., K, n_gpus) at first use, and the fixed-point solution for B* should be described in one sentence for reproducibility.","section":"Section 3.2, Eq. for B*"},{"comment":"The left panels of Fig. 8 use the y=0 line as static baseline; the caption says normalized to same-window static but the DSv3 EP16 panel appears to show different scales. Please clarify the normalization and the number of windows for each panel.","section":"Section 5.10, Fig. 8"},{"comment":"The paper uses the phrase model-scored extensively but not always in the abstract and introduction; adding a sentence early on that the headline phase-diagram numbers are simulated under the calibrated model would improve clarity.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is substantively strong and unusually honest about its limitations, but the mixed-regime statistic in Fig. 5 appears to contain a genuine inconsistency with the paper's own n* values, and the identifiability of the deployment-time n* is weak. These issues affect the central motivational claim and the precision of the phase diagram, though they are fixable by recomputation and sensitivity analysis. I recommend major revision rather than rejection because the core model, the solver, and the wall-clock validation are sound, and the authors have shown willingness to state limitations explicitly. The scale extrapolations to EP32-64 are model-scored and should be presented with clear caveats, as the paper already does in L1-L2."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has real content: a measured max-affine cost model for expert-parallel MoE, a tile-aware extension, NP-hardness at dispatch granularity, a sensible phase diagram, and an unusually honest evaluation that includes negative results and explicit model-space caveats. The microbenchmark fits (4-8% error), the roofline check tying b to HBM weight streaming, and the like-for-like LPLB comparison are all solid work. I also credit the authors for reporting the identifiability ridge in the full-pipeline fit and for not hiding that the largest wins are extrapolations.\n\nThe soft spot is the central motivation. Section 2.4 claims that 92-100% of decode batches at 128-512 tokens/GPU contain both regimes at once, with hot experts in the linear region. With the stated n* = 156-168 tokens, no expert at B=128 can have more than 128 tokens, so 0% of tokens can be linear. The statistic cannot have been computed with the headline inflection. The full-pipeline fit implies n* around 46, but that quantity is not identifiable from the pipeline logs (b in [5.1, 18.8], corr(a,b) = -0.997), and the paper appears to conflate the kernel inflection with a deployment-time quantity. This matters because the \"regimes coexist inside single batches\" claim is the stated reason no fixed proxy is safe, and the phase-diagram mixed zone is where the largest extrapolated wins live. The wall-clock flat-regime results and the end-to-end serving wins on Qwen3-235B are not invalidated, but their interpretation as mixed-regime effects is now questionable.\n\nI would not desk-reject this. The cost model, the solver, and the empirical methodology are valuable even if the mixed-regime prevalence is overstated. The authors need to redo the Fig. 5 statistic with the correct inflection (or state explicitly which n* they used), reconcile it with the full-pipeline identifiability analysis, and adjust the narrative about decode batches straddling the hardware inflection. Absent that fix, the phase diagram should be presented as model-scored extrapolation with a weaker motivational claim. A serious referee would catch this quickly; the paper deserves that scrutiny rather than a desk rejection.","headline":"A useful two-regime cost model and a well-built dispatcher, but the paper's headline mixed-regime statistic is arithmetically impossible under its own inflection point.","tokens_in":28985,"tokens_out":2424,"would_cite":true,"duration_ms":29058,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that per-expert fp8 grouped-GEMM time is flat up to about 156–168 tokens and linear above it, and that a makespan-aware dispatcher over the resulting two-regime cost model stays within about 1% of the best fixed policy…","keywords":["mixture-of-experts","expert parallelism","load balancing","makespan","grouped GEMM","memory-bound","phase diagram","fp8 inference"],"falsifier":"Measure per-layer wall-clock MoE-block makespan on a 32- or 64-GPU deployment across a batch-size ladder from 32 to 2048 tokens per GPU and compare it with the model's predicted $t_g=\\max(a+bG_g,\\,c+\\beta N_g)$; if the flat-to-linear inflection near 156–168 tokens per expert, or the claimed 92–100% fraction of decode batches containing both regimes, fails to appear on another kernel generation or another routing distribution, the central claim collapses.","tokens_in":27674,"feed_emoji":"⚖️","tokens_out":8441,"duration_ms":77617,"temperature":0.7,"pith_summary":"What the paper tries to establish is that the standard proxies used to balance expert-parallel MoE serving — token counts and activated-expert counts — both misprice the hardware cost, and that a makespan-aware dispatcher over a measured two-regime cost model is the right objective. Measurements on two GPU generations show per-expert fp8 grouped-GEMM time is flat up to about 156–168 tokens and linear above it; the flat regime is HBM weight streaming, the linear regime is token compute rounded to 128-token tiles. On recorded decode batches, hot experts sit in the linear regime and cold experts in the flat regime at the same time, so no fixed proxy is safe. The paper formalizes per-batch dispatch as a fixed-charge makespan problem and presents TEMPO, which solves it in milliseconds and stays within about 1% of the best fixed policy at every phase-grid point while winning up to 15.5% where regimes mix. A phase diagram, not a universal win, is the claim.","feed_headline":"Expert time is flat then linear; makespan dispatch wins up to 15.5%","feed_subtitle":"A max-affine two-regime cost model lets a dispatcher track the best fixed policy within 1 percent.","key_machinery":"The load-bearing object is the two-piece max-affine cost profile $t_g=\\max(a+bG_g,\\,c+\\beta N_g)$; $b$ is the per-replica activation floor (measured at 1.13–1.24$\\times$ the HBM roofline for the expert's weights), $\\beta$ is the linear per-token compute term, and the tile-aware extension adds $b_2(T-G)$ with $T=\\sum_e \\lceil n_e/128\\rceil$. On top of this, dispatch is cast as a fixed-charge makespan problem (minimize $\\max_g \\max(a+b\\sum_e z_{e,g},\\,c+\\beta\\sum_e x_{e,g})$), which is NP-hard on two fully replicated GPUs yet polynomial in each degenerate limit ($b\\to0$ reduces to the token LP; $\\beta\\to0$ to semi-matching). The practical solver — cost-aware seeding, augmenting-chain activation rebalancing, bottleneck local search with partial splits, and an ensemble with a 1% switching tolerance — solves it in about 2 ms off the critical path and inherits an additive approximation guarantee via a round-robin whole-expert placement.","core_discovery":"The central discovery is that per-GPU MoE-block time obeys a max-affine law, $t_g=\\max(a+bG_g,\\,c+\\beta N_g)$, where $G_g$ is the number of activated expert replicas and $N_g$ is the total token count. Below the inflection $n^*\\approx156$–$168$ tokens per expert, time is essentially flat in tokens because streaming the expert's weights from HBM dominates; above it, grouped GEMM rounds each expert's tokens up to 128-token M-tiles, so fragmenting an expert manufactures padded compute. Real decode batches mix both regimes simultaneously (92–100% of recorded batches), which is why a dispatcher optimizing the max of the two terms, i.e., the makespan, can beat both token-only and activation-only policies. The same measured shape transfers to a newer GPU generation and is captured with one extra parameter for the full tile staircase, though the largest claimed gains (up to 15.5%) are model-scored extrapolations to EP32–64.","pith_inferences":["The same two-regime mechanism should appear on other GPU generations and fp8 kernels, but with different $b$, $\\beta$, and tile heights, so the phase diagram is a template to be recalibrated per hardware, not a universal constant.","Monitoring per-GPU activation counts and tile counts in a live server could let an operator detect which regime the current batch is in and switch dispatcher mode without running a solver.","The cost model prices weight movement itself (the activation floor is the cost of touching an expert's weights), which suggests that token-moving dispatch is the only zero-marginal-cost lever at decode scale.","The NP-hardness at dispatch granularity hints that placement and dispatch should be co-designed: a placement that minimizes the variance of activation floors would shrink the region where dispatch must act at all."],"forward_implications":["If the two-regime cost shape is correct, all token-only and activation-only dispatchers are systematically wrong outside their home regime, with modeled block-time spreads of 1.4–1.6$\\times$ (p95 up to 1.7$\\times$) between proxy dispatches of the same batch.","A deployment can locate itself on a phase diagram rather than guess: the policy flip boundary $B^*$ is analytically predictable from expert popularity, replication, and batch size, and it lands in the observed flip band in 12/12 grid columns.","Inside the win region, makespan-aware dispatch delivers end-to-end gains (4–6% throughput and roughly a 15.6% p99 latency cut on one flagship model) while staying within 1% of the best fixed policy elsewhere.","The dispatch layer repairs the latency tail and makes tight replica budgets safe to exploit, but it cannot repair the mean: under stale placement, throughput loss is a provisioning error that only the placement layer can fix.","On multi-node expert parallelism, source-to-replica pairing is part of the cost model: a topology-aware split flips a 3.5% regression into a 4–7% gain at the all-to-all-bound point."],"supporting_citations":[{"why":"Provides the fp8 grouped-GEMM kernel whose measured per-expert timing produces the flat-then-linear shape.","marker":"[4]"},{"why":"Supplies the placement-layer replica sets that bound the dispatch problem and the uniform-split baseline used throughout.","marker":"[5]"},{"why":"Defines the token-count linear-program dispatch objective and explicitly leaves nonlinear expert cost open, the gap this paper closes.","marker":"[6]"},{"why":"Defines activation-count balancing, the strongest fixed proxy in the flat regime and a key baseline.","marker":"[13]"},{"why":"Provides the semi-matching augmenting-path algorithm used by the activation-rebalancing stage and the polynomial algorithm in the beta-to-zero limit.","marker":"[10]"},{"why":"Supplies one flagship expert shape and routing statistics used to calibrate the cost model and set the win-region experiments.","marker":"[2]"},{"why":"Provides the splitting-jobs scheduling view used to reason about the degenerate limits of the dispatch problem.","marker":"[18]"}],"fun_headline_variants":["Two-regime MoE cost model: makespan dispatcher beats fixed policies by 15.5%","Flat then linear: MoE expert time needs a makespan dispatcher, not token counts","Max-affine MoE cost: dispatcher beats baselines up to 15.5% in mixed regimes","Expert time is not linear in tokens: makespan-aware dispatch wins 15.5%","MoE makespan dispatch: two regimes, one cost model, up to 15.5% gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The biggest wins rest on the assumption that the two-regime cost model calibrated on one 8-GPU testbed transfers to full serving pipelines and to the larger expert-parallel sizes (EP32–64) where the headline gains are extrapolated, with the fitted traffic term as the least independently validated part.","fun_headline_variants_meta":{"raw":{"variants":["Two-regime MoE cost model: makespan dispatcher beats fixed policies by 15.5%","Flat then linear: MoE expert time needs a makespan dispatcher, not token counts","Max-affine MoE cost: dispatcher beats baselines up to 15.5% in mixed regimes","Expert time is not linear in tokens: makespan-aware dispatch wins 15.5%","MoE makespan dispatch: two regimes, one cost model, up to 15.5% gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001263,"raw_usage":{"total_tokens":5288,"prompt_tokens":1181,"completion_tokens":4107,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":797,"completion_tokens_details":{"reasoning_tokens":3981}},"tokens_in":797,"tokens_out":4107,"duration_ms":27097,"temperature":1.0,"reasoning_tokens":3981,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:40:41.170748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure per-layer wall-clock MoE-block makespan on a 32- or 64-GPU deployment across a batch-size ladder from 32 to 2048 tokens per GPU and compare it with the model's predicted $t_g=\\max(a+bG_g,\\,c+\\beta N_g)$; if the flat-to-linear inflection near 156–168 tokens per expert, or the claimed 92–100% fraction of decode batches containing both regimes, fails to appear on another kernel generation or another routing distribution, the central claim collapses.","supporting_citations":[{"cited_title":"DeepGEMM: clean and efficient FP8 GEMM kernels","cited_arxiv_id":null,"evidence_quote":"Provides the fp8 grouped-GEMM kernel whose measured per-expert timing produces the flat-then-linear shape."},{"cited_title":"EPLB: Expert parallelism load balancer","cited_arxiv_id":null,"evidence_quote":"Supplies the placement-layer replica sets that bound the dispatch problem and the uniform-split baseline used throughout."},{"cited_title":"LPLB: An LP-based load bal- ancer for expert parallelism","cited_arxiv_id":null,"evidence_quote":"Defines the token-count linear-program dispatch objective and explicitly leaves nonlinear expert cost open, the gap this paper closes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the semi-matching augmenting-path algorithm used by the activation-rebalancing stage and the polynomial algorithm in the beta-to-zero limit."},{"cited_title":"Parallel machine scheduling with splitting jobs.Discrete Applied Mathe- matics, 103(1-3):259–269, 2000","cited_arxiv_id":null,"evidence_quote":"Provides the splitting-jobs scheduling view used to reason about the degenerate limits of the dispatch problem."}],"review_version":1}