{"id":"d5e2f613-3104-4209-a8fc-55047646ddd4","arxiv_id":"2508.19373","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HAP uses ILP over module-specific hybrid parallel choices to speed up MoE inference, reporting up to 1.77x versus tensor parallelism on tested GPUs.","lead":"HAP is a system for Mixture-of-Experts (MoE) large language model inference that picks, for each GPU and each inference phase, how to split attention and expert modules, using an Integer Linear Programming search guided by fitted latency models. It reports up to 1.77x faster inference than a tensor-parallel baseline on A100/A6000/V100 nodes, but no code or data are released and gains shrink to near-parity in decoding-heavy workloads.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Simulator ranking fidelity unvalidated; additive per-module latency model may misrank hybrid configurations, so reported speedups over TP are not yet supported as general.","rationale":"The reader's weakest_assumption correctly identifies the additive latency decomposition and simulator accuracy as the most load-bearing element. My stress-test agrees, but sharpens it: the real risk is not just raw prediction error but ranking fidelity, because the ILP makes decisions based on comparisons of predicted totals. A 10% error in component prediction can flip the chosen configuration if two candidates are within 10% in true latency. The paper never measures whether the ILP-selected configuration is actually the true optimum or within a small margin of it. I considered other concerns—the arbitrarily pruned search space (DP+EP+TP excluded for experts), the uncertain transition-cost formula (Eq. 6), and the absence of a static-hybrid baseline—but these would only weaken the incremental contribution of HAP's adaptive component; they do not directly threaten the claimed superiority over TP if the simulator ranking is correct. The exhaustive-enumeration test would settle the ranking question decisively. The verdict remains CONDITIONAL: the paper's central claim is plausible but not fully supported without this validation.","tokens_in":11338,"tokens_out":8782,"duration_ms":98190,"concrete_test":"Run Mixtral-8x7B on 4xA6000 with 4096-token prompt, batch size 32 (and 4xA100 with same config). Enumerate every valid combination in HAP's own search space (attention DP/TP/DP+TP x expert TP/EP/TP+EP, with prefill and decode strategies coupled as allowed by Eq. 5) using the same DeepSpeed-FastGen harness, and measure end-to-end latency for each. Compare the actual latency of the ILP-selected configuration to the empirically best configuration. Repeat at batch sizes 8 and 64. If HAP's chosen config is >5% slower than the best in any tested case, the simulator ranking fidelity is insufficient to support the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The ILP objective (Eq. 4) minimizes a predicted total formed by adding independently simulated attention, expert, and communication latencies (Eqs. 1-3). This additive decomposition assumes no overlap or contention between modules. In real MoE inference, expert-parallel All-to-All can overlap with attention compute (as DeepEP demonstrates), and memory bandwidth contention couples the modules. The random forest correction terms eta and rho are fit to isolated operator latencies, so they cannot capture configuration-dependent interactions. The paper reports component errors <5-10% but never validates the critical property: that the predicted total preserves the true ranking of all candidate configurations. If the simulator systematically biases communication-heavy strategies (e.g., overestimating TP AllReduce cost), the ILP will favor EP/hybrid configurations even when TP is actually optimal, inflating the reported speedups. This risk is highest exactly for the long-context constrained-output scenarios that produce the headline 1.68-1.77x gains, where prefill communication dominates. Without an exhaustive-search ranking check, the central claim that HAP 'determines' optimal or near-optimal configurations is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HAP, a system that automatically selects per-module hybrid parallel strategies for MoE LLM inference. It decomposes a transformer layer into Attention and Expert modules, estimates per-module compute and communication latencies using FLOP/bandwidth formulas corrected by random-forest regressors, enumerates DP/TP for the Attention module and EP/TP for the Expert module, and solves an ILP to choose prefill and decode strategies, including a transition-cost model. A quantized CPU-side backup of expert weights is used to reduce the cost of changing the Expert module's parallelization between phases. Experiments with DeepSpeed-FastGen on Mixtral-8x7B, Qwen1.5-MoE-A2.7B, and Qwen2-57B-A14B across A6000, A100, and V100 platforms report speedups over a TP baseline of up to 1.77x, strongest in long-context / constrained-output scenarios.","tokens_in":11699,"tokens_out":5929,"duration_ms":67007,"significance":"If the search and latency simulation are faithful, HAP is a practical, scenario-adaptive alternative to static TP/EP for MoE inference, with modest search overhead (ILP solved in <1 s, and that time is included in the reported end-to-end latencies) and a concrete strategy-transition mechanism. The evaluation is broad, spans heterogeneous interconnects and model families, and honestly reports configurations where TP remains preferable. The main risk is that the central 'optimal or near-optimal strategy' claim rests on simulator ranking fidelity, which is not directly validated. The contribution is therefore conditional on additional validation rather than fundamentally unsound.","major_comments":[{"comment":"The ILP as written is not a well-defined mathematical program: S_k and E_i are one-hot vectors, while E_t, E_d, E_e are scalar degrees; the objective uses T_C^{ki} and T_C^{kj} without definition; and the constraint N = A_t × A_d = E_d × E_t × E_e is ambiguous because E_d appears both as a scalar degree and, via E_i, in the one-hot strategy notation. Please rewrite with consistent notation for scalar degrees and strategy indicators, and define T_C(i,j) explicitly.","section":"Section III-C, Eq. (4)-(5)"},{"comment":"Component-level relative errors of 5-10% (Fig. 5) do not establish that the additive sum in Eqs. (1)-(3) preserves the true ranking of candidate configurations. A systematic bias in, say, the AllReduce model would systematically favor communication-light hybrids and inflate the reported speedups. Please add an exhaustive-search ranking check for at least one model/GPU/scenario: enumerate every feasible (Attention, Expert) strategy pair, compare measured vs. predicted end-to-end latency, and report rank correlation and the gap between HAP's selected latency and the measured best. Also test the additivity assumption by measuring whether isolated module latencies combine without overlap or memory-bandwidth contention.","section":"Section IV-B vs. Section III-B"},{"comment":"The '2× baseline TP activation footprint' upper bound for EP activation memory is an unvalidated ad-hoc multiplier. It enters the memory constraint and can change which configurations are feasible, and thus the ILP optimum. Provide empirical activation-memory measurements under EP with realistic load imbalance, or a sensitivity analysis showing that the ILP solution is invariant for a plausible range of this multiplier (e.g., 1.0× to 2.5×).","section":"Section III-A2, Eq. (5)"},{"comment":"The pruning of DP+EP+TP combinations for the Expert module is asserted to be 'generally suboptimal' based on 'existing experience' but no evidence is given. Because the paper claims to search for the optimal configuration, this pruning needs justification, e.g., a benchmark subset showing that pruned configurations are always dominated by unpruned ones. Otherwise the claim should be restricted to the unpruned search space.","section":"Section III-C"},{"comment":"The empirical section reports no run-to-run variance or error bars, and many headline claims are 'up to' maxima over batch sizes. Report mean ± std over multiple repeated runs for representative configurations and report median (typical) speedups in addition to maxima. Also reconcile the abstract platform ordering (1.68×/1.77× on A100/A6000) with Section IV-C3, where 1.68× is reported on A6000 and 1.77× on A100.","section":"Section IV"}],"minor_comments":[{"comment":"Typos: 'implys' in Section III-B; 'are are' in Section III-C; 'dopted' in Section III-D. Please proofread.","section":"Throughout"},{"comment":"Specify how many samples were used to train/test the random-forest correctors and whether the test set is disjoint from the configurations and batch sizes used in the end-to-end experiments.","section":"Section IV-B"},{"comment":"The dequantization-latency dictionary V_dequant → T_dequant is introduced but its construction and measurement are not described. Add a sentence on how this dictionary was obtained and how sensitive C_ij is to its accuracy.","section":"Eq. (6)"},{"comment":"The comparison baseline is TP only. Since DeepEP and other EP-based systems are cited as related work, a direct comparison against a state-of-the-art EP or static-hybrid baseline would strengthen the positioning, even if only for the long-context constrained-output scenario.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The paper does not appear to provide code or an artifact link, which limits reproducibility of the measured speedups. If the authors can supply the exhaustive ranking-validation experiment and clarify the ILP notation, I would be willing to reconsider the recommendation. The topic fits the journal's scope and the practical contribution is potentially useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid, well-scoped systems paper that deserves a real referee, but the headline speedups are not yet as general as they look. The new thing is not any single technique—ILP search, cost models, quantized CPU backup all exist—but the combination: decomposing an MoE layer into attention and expert modules, letting each pick a different parallel strategy per phase (prefill vs decode), and paying for the switch with an INT4 CPU-side backup streamed in during prefill. The evaluation is broader than most: three model families, three GPU types, four scenario types, and the long-context constrained-output case shows a genuine phase-specific win (1.68x/1.77x/1.57x). The paper is honest about the short-context extended-output case where HAP often lands on TP and gains almost nothing. That is a good sign.\n\nSoft spots are real but not fatal. First, the simulator's ranking fidelity is never validated. The paper reports per-operator error <5–10% on computation and communication latencies, but the ILP objective is a sum of independently modeled attention, expert, and communication latencies with no overlap or contention terms. In practice attention compute can overlap with all-to-all, and memory bandwidth couples the modules, so the predicted total can misrank configurations even when each component is well calibrated. That is exactly the risk for the long-context prefill-heavy cases that produce the biggest speedups. A simple check—enumerate all candidate strategies, compare predicted vs measured end-to-end latency, show the ranking—would have answered this. Second, the EP activation memory bound is a single sentence (\"doubles the baseline TP activation footprint\") with no derivation or sensitivity analysis; it is load-bearing for pruning the search space. Third, the ILP equations have notation problems: the objective in Eq. (4) uses S, E, R but the constraint block introduces Ad, Ed, Et, Ee inconsistently, and the divisibility constraints are under-specified. A careful reader can mostly reconstruct the intent, but the formalization is not clean. Finally, no code or data is released, and the plots have no error bars, so the measured speedups are hard to independently reproduce.\n\nOverall: the central claim—that phase-specific hybrid parallelism can beat a static TP baseline on real MoE models—is plausible and the experiments support it for the configurations tested. The generalization to unseen hardware or batch sizes rests on the unvalidated simulator. This is worth a serious referee; I would ask for code/data, a ranking-fidelity experiment, and a cleaned-up ILP formulation. If those land, this is a useful result for anyone building MoE serving stacks.","headline":"Phase-specific hybrid parallelism for MoE inference is a real and useful system contribution, but the unvalidated latency-simulator ranking and missing code/data keep the headline speedups from being fully supported yet.","tokens_in":12123,"tokens_out":2587,"would_cite":true,"duration_ms":26111,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HAP argues that Mixture-of-Experts inference should use different parallel strategies for attention and expert modules, and for prefill versus decode, picked by an ILP solver that minimizes simulated end-to-end latency.","keywords":["mixture-of-experts inference","hybrid parallelism","adaptive parallelism","integer linear programming","latency simulation","tensor parallelism","expert parallelism","LLM serving"],"falsifier":"Run HAP and a full factorial sweep of all feasible hybrid configurations on a new GPU, batch-size, and context-length combination and compare actual end-to-end latencies: if the ILP-selected configuration is beaten by another configuration by more than the simulation model's reported error margin, the additive latency decomposition or the fitted simulators are not sufficient. A second check: measure the transition between prefill and decode strategies directly; if asynchronous CPU-side dequantization does not overlap with the first decode layers and the switching cost exceeds the predicted C_i","tokens_in":11291,"feed_emoji":"⚙️","tokens_out":8076,"duration_ms":80258,"temperature":0.7,"pith_summary":"Mixture-of-Experts (MoE) inference is usually served with one static parallel strategy, typically tensor parallelism, applied uniformly to every layer and both inference phases. HAP argues that this is systematically suboptimal: attention computation, expert computation, and collective communication respond differently to parallelism depending on batch size, context length, and interconnect bandwidth. The paper's central proposal is to decompose each MoE layer into an Attention module and an Expert module, build lightweight latency simulators for each, then let an integer linear program pick the best hybrid strategy for prefill and decode separately, with a cheap transition between phases. If the reported results hold, the payoff is practical: up to 1.68x, 1.77x, and 1.57x speedups over a tensor-parallel baseline on A100, A6000, and V100 nodes, with no change to the model itself. The authors also claim the search overhead is under a second and is included in all reported latencies.","feed_headline":"Hybrid per-module parallelism speeds MoE inference up to 1.77x","feed_subtitle":"HAP solves the best attention/expert parallel config with ILP, cutting prefill and decode latency on A100, A6000, and V100.","key_machinery":"The central mechanism is HAP's two-level decomposition plus ILP search: (1) module-specific latency estimators T_cal = (FLOPs / peak FLOPs/s) * eta and T_comm = (data volume / bandwidth) * rho, where eta and rho are fitted with a random-forest regression on benchmarked operation latencies; (2) a hierarchical search space over Attention strategies (DP, TP, or DP+TP) and Expert strategies (EP, TP, or EP+TP); and (3) an ILP objective that minimizes N_layer times [prefill cost + S_output times decode cost] plus a switching-cost term E_i^T C_ij E_j, with divisibility and memory constraints. The switching cost is itself minimized by comparing direct AllGather/AllToAll weight redistribution against","core_discovery":"The paper establishes that the optimal parallel configuration for MoE inference is not a single strategy but a phase- and module-dependent combination. Its central discovery is that, by splitting the MoE layer into attention and expert modules and simulating each module's compute and communication latency (a FLOP-count model corrected by a random forest, and a bandwidth model similarly corrected), the search over hybrid parallel configurations can be formulated as an integer linear program whose objective is predicted end-to-end latency including the cost of switching strategies between prefill and decoding. The experiments then show HAP-chosen configurations match or beat the standard tenso","pith_inferences":["The same module-decomposition plus ILP search could be applied to other sparse or hybrid architectures, such as models with grouped-query attention and shared experts, by swapping in new per-module simulators.","The prefill/decode split is a coarse two-phase approximation; continuous batching and chunked prefill introduce more than two phases, and the framework could be extended to search over a schedule of configurations rather than one transition.","The quantized-CPU-backup transition is a special case of hiding configuration changes behind computation; one could generalize it to online model re-sharding or to swapping experts in and out during serving.","If the random-forest corrections are recalibrated on new hardware, the approach should transfer to other interconnects and GPU generations, but transferability across batch sizes and hardware is an empirical claim the paper tests only on three platforms."],"forward_implications":["A single inference system can switch its expert-module parallelism between prefill and decode, getting EP-style prefill latency and TP-style decode latency with small transition overhead.","In communication-bound workloads, choosing low-communication strategies for both modules is what produces the largest reported wins (1.57x to 1.77x).","The search itself costs less than a second for 4- and 8-GPU nodes, and because that time is included in reported end-to-end latency, the optimizer does not have to be amortized over long runs.","The same solver can handle MoE designs with few large experts or many small or shared experts by changing only the module latency inputs, which is why the reported gains appear across different MoE architecture families.","When decoding dominates, the solver picks tensor parallelism, so HAP should not be much worse than the TP baseline even in regimes where TP is already optimal."],"supporting_citations":[{"why":"Host inference framework whose tensor-parallel execution serves as the baseline in every reported speedup.","marker":"[7]"},{"why":"Defines the sharding and all-to-all communication patterns that constitute the parallel-strategy search space.","marker":"[9]"},{"why":"Supplies the sparse expert architecture used in the primary evaluation and runtime breakdowns.","marker":"[11]"},{"why":"Provides evidence that expert parallelism suffers load imbalance and inefficiency, motivating hybrid rather than pure EP.","marker":"[17]"},{"why":"Gives the static hybrid data/tensor/expert parallel serving setup that HAP contrasts with and improves upon.","marker":"[18]"},{"why":"Shows an ILP-based parallelism search; HAP builds on its formulation while including inference latency rather than communication only.","marker":"[21]"},{"why":"Introduces switchable parallelism for MoE; HAP's dynamic transition mechanism is the inference-oriented adaptation.","marker":"[22]"}],"fun_headline_variants":["ILP-chosen parallel splits speed MoE inference 1.77x","Adaptive MoE parallel configs rival tensor parallelism","Split attention and expert modules for faster MoE inference","HAP selects per-module parallel configs to cut MoE latency"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that a layer's total latency is the sum of independently simulated attention, expert, and communication latencies; if memory-bandwidth contention, kernel overlap, or hardware-specific coupling breaks that additivity, the configuration that minimizes the simulated sum will not minimize real end-to-end latency.","fun_headline_variants_meta":{"raw":{"variants":["ILP-chosen parallel splits speed MoE inference 1.77x","Adaptive MoE parallel configs rival tensor parallelism","Split attention and expert modules for faster MoE inference","HAP selects per-module parallel configs to cut MoE latency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3184,"prompt_tokens":761,"completion_tokens":2423,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":2360}},"tokens_in":505,"tokens_out":2423,"duration_ms":19848,"temperature":1.0,"reasoning_tokens":2360,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:47:59.306606+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HAP and a full factorial sweep of all feasible hybrid configurations on a new GPU, batch-size, and context-length combination and compare actual end-to-end latencies: if the ILP-selected configuration is beaten by another configuration by more than the simulation model's reported error margin, the additive latency decomposition or the fitted simulators are not sufficient. A second check: measure the transition between prefill and decode strategies directly; if asynchronous CPU-side dequantization does not overlap with the first decode layers and the switching cost exceeds the predicted C_i","supporting_citations":[{"cited_title":"Deepspeed-moe: Advancing mixture- of-experts inference and training to power next-generation ai scale,","cited_arxiv_id":null,"evidence_quote":"Gives the static hybrid data/tensor/expert parallel serving setup that HAP contrasts with and improves upon."},{"cited_title":"Alpa: Automating inter-and {Intra- Operator} parallelism for distributed deep learning,","cited_arxiv_id":null,"evidence_quote":"Shows an ILP-based parallelism search; HAP builds on its formulation while including inference latency rather than communication only."},{"cited_title":"Tutel: Adaptive mixture-of-experts at scale,","cited_arxiv_id":null,"evidence_quote":"Introduces switchable parallelism for MoE; HAP's dynamic transition mechanism is the inference-oriented adaptation."}],"review_version":1}