{"id":"c2750f9e-4efd-413f-a90b-f3ba7d3a628d","arxiv_id":"2608.06723","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hybrid analytical-ML framework predicts LLM inference latency and energy from architectural parameters, with MAPE below 5 percent on selected models and about 10 percent on a broad set.","lead":"HYMELL is a three-level hybrid model that predicts LLM inference latency and energy by combining regression-based analytical operators with machine-learning corrections. It reports under 5 percent error for several common models, enabling hardware-free design space exploration.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing family-level train/test split makes the claimed under-5% accuracy unverified: the reported LLaMA 3 8B result may be interpolation over the profiled architectural distribution rather than generalization to unseen architectures.","rationale":"I agree with the reader's conditional verdict: the paper is a plausible empirical framework whose strongest generalization claims need stronger evidence. However, I consider the GEMM regime-threshold issue to be secondary. The hierarchical design deliberately places MLPs above the analytical operators, and those MLPs receive both operator-cost predictions and architectural parameters, so they can learn to correct systematic operator errors for shapes encountered in training. The more load-bearing assumption is that the end-to-end MLP predicts cost for architectures it has not seen, and the paper does not provide the family-level split needed to demonstrate that. Table 7's LLaMA 3 8B numbers could be in-sample or near in-sample; the abstract and conclusion elevate them as the headline result, while Table 6's broader average is roughly 10%, a noticeable gap. A leave-one-family-out test would settle whether HYMELL is a genuine architectural predictor or a high-dimensional interpolator. This does not change the reader's verdict direction, but it sharpens the condition: release data with a family-excluded validation, not merely code. I therefore set verdict_should_be to CONDITIONAL and agreement_with_reader to partial because the reader's identified weakest assumption is plausible but not, in my reading, the single most load-bearing one.","tokens_in":12213,"tokens_out":8093,"duration_ms":83098,"concrete_test":"Release the full profiling dataset and exact train/test split, then run a leave-one-family-out re-evaluation: exclude every configuration belonging to the LLaMA 3 8B family from block-level and end-to-end MLP training, retrain with the same hyperparameters and thresholds, and report per-sequence MAPE for LLaMA 3 8B prefill and decode. Repeat independently for the Qwen MoE family. If LLaMA 3 8B MAPE stays below about 5% and the dense/MoE averages remain near 10%, the generalization claim is supported; if MAPE jumps to the 15-20% range or higher, the reported accuracy is in-distribution interpolation and the central claim should be downgraded.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that HYMELL predicts end-to-end latency and energy from architectural parameters, enabling hardware-free design space exploration. That claim depends on the end-to-end MLP (Section 3.6) generalizing beyond its ~1,200 profiled configurations. Section 5.1 reports a held-out evaluation (Table 6) and Table 7 reports under-5% error for LLaMA 3 8B, but the paper never specifies how the held-out set was constructed or whether LLaMA 3 8B, Mistral 7B, and Qwen MoE were excluded from training. Because the dataset varies model dimension, layers, heads, sequence length, and batch size, the tested models may lie inside the convex hull of the training distribution. If so, the highlighted under-5% result demonstrates interpolation, not the architecture-level extrapolation promised in the abstract and conclusions, and even Table 6's ~10% average may be optimistic for configuration families absent from training. The GEMM regime-threshold concern raised by the reader is real (Table 3 shows 9.4% MAPE in the balanced compute-dominant regime), but the upper-level MLPs can partially absorb operator-level residuals for shapes seen during training; the unresolved risk is behavior for unseen architectural shapes, which is exactly what the design-space-exploration claim requires.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HYMELL, a three-level hybrid framework for predicting latency and energy of LLM inference on GPUs. Level 1 consists of 'analytical' estimators for RMSNorm, softmax, and GEMM, which are actually polynomial regressions with features motivated by workload and memory traffic. Level 2 uses lightweight MLPs to predict attention and FFN block costs from the Level-1 outputs plus architectural parameters. Level 3 combines block-level predictions with global model parameters in an MLP to estimate end-to-end prefill and decode latency/energy. Evaluation on an NVIDIA H100 reports operator-level MAPE of 1.7%-6.4%, block-level MAPE of 0.4%-6.3%, and end-to-end MAPE of about 10-13%, with under 5% error for LLaMA 3 8B at batch size 1. A cross-device experiment on an RTX A6000 and an ablation on linear attention are also included. The central claim is that HYMELL enables fast, hardware-free design-space exploration by predicting execution costs directly from architectural parameters.","tokens_in":12549,"tokens_out":5389,"duration_ms":48762,"significance":"If the reported accuracy generalizes beyond the profiled configurations, HYMELL would provide a useful surrogate for LLM inference cost estimation, supporting design-space exploration without repeated hardware measurements. The hierarchical decomposition into operator, block, and end-to-end levels is a sensible way to combine low-level kernel behavior with high-level architectural parameters, and the inclusion of dense and MoE FFNs, MHA and GQA, and both prefill and decode phases is a strength. The cross-device experiment (Table 8) and linear-attention ablation (Table 9) extend the evidence base beyond a single architecture/hardware combination. However, the validation methodology is not yet rigorous enough to establish the generalization claim: the construction of the held-out end-to-end set is not described, the operator-level estimators lack a train/test split, and no measure of variability is given for any of the reported metrics. The paper does not mention code or data release, which would also help reproducibility.","major_comments":[{"comment":"The operator-level 'analytical' estimators are polynomial regressions whose coefficients are fitted to measured data, but the manuscript never states whether the MAPE and R^2 values in Tables 1-3 are computed on the training data, a validation split, or held-out test data. Since these estimates are consumed by the upper-level MLPs, any in-sample optimism at Level 1 propagates upward and undermines the end-to-end generalization claim. The paper must specify the train/test split or cross-validation procedure for each operator estimator and report out-of-sample metrics, ideally with variance across splits.","section":"Section 5.1, Eqs. (1)-(6), Tables 1-3"},{"comment":"The end-to-end evaluation is described as being on 'held-out LLM architectures' (Table 6) and specific models (Table 7, Figure 4), but the paper does not explain how the held-out set was constructed, whether LLaMA 3 8B, Mistral 7B, and Qwen MoE were excluded from the ~1,200-configuration training set, or whether the tested models lie outside the architectural distribution covered by training. Without this information, the under-5% per-model results may simply reflect interpolation over the profiled families rather than the architecture-level extrapolation promised in the abstract and conclusions. The authors should describe the held-out split in detail and, ideally, evaluate on architecture families not present in the training set.","section":"Section 5.1-5.2, Tables 6-7, Figure 4"},{"comment":"The GEMM regime partition depends on five thresholds (F_thr, AI_low, AI_mid, AI_high) whose values are reported in Section 5.2, but no procedure for selecting these thresholds is given. If the thresholds were tuned on the same data used to evaluate the estimator, the regime-specific MAPE values (including the 9.399% for the balanced compute-dominant regime) may be optimistically biased, and the model may not generalize near regime boundaries where cuBLAS kernel switches occur. The authors should state how the thresholds were chosen (e.g., via a validation set or prior knowledge) and assess sensitivity to these values.","section":"Section 3.3, Table 3, Section 5.2"},{"comment":"None of the reported MAPE or R^2 values include error bars, confidence intervals, or repeated-training-seed variation. Given that all MLPs use stochastic optimizers and the datasets are finite, some variance across seeds and data splits is expected, and this variance is necessary to judge whether differences such as dense vs. MoE end-to-end MAPE are meaningful. The authors should add repeated-seed or repeated-split statistics, or at least report the range across a small number of runs.","section":"Section 5.2, Tables 1-9, Figure 4"}],"minor_comments":[{"comment":"The AMALI values in Table 7 are aggregate MAPE figures repeated across sequence lengths, not per-sequence measurements. The text acknowledges this, but the table layout may still mislead readers. Please add an explicit note in the table caption that AMALI does not report per-sequence errors or energy predictions.","section":"Section 5.2, Table 7"},{"comment":"The conclusion states that HYMELL achieves 'less than 5%' error for Llama3, Mistral, and Qwen models, but Table 7 reports numerical error only for LLaMA 3 8B; the claims for Mistral and Qwen rest on the visual representation in Figure 4, which does not show exact per-sequence values or error bars. Provide a table of per-model, per-sequence errors, or clarify that the figure is qualitative.","section":"Section 5.2, Figure 4 and Conclusions"},{"comment":"The feature set for the GEMM estimator is said to have been 'obtained via ablation', but the ablation procedure and the marginal contribution of each term are not reported. Including an ablation table or a brief description of the selection criterion would improve interpretability and support the claim that all retained features are significant.","section":"Section 3.3, Eqs. (5)-(6)"},{"comment":"The interpretation of log2(d_model) and N*log2(d_model) as 'tree-based reduction' depth is plausible for reductions but may not hold for all implementations of RMSNorm on modern GPUs. A brief justification or reference would help, since the regression coefficients absorb whatever the kernel actually does.","section":"Section 3.1, Eqs. (1)-(2)"},{"comment":"The measurement methodology says each configuration was repeated until cumulative execution time exceeded 10 seconds, then later says large prefill workloads used only five repetitions. It would be helpful to state explicitly how many repetitions were used for the operator, block, and end-to-end datasets, and whether the 10-second rule applied uniformly.","section":"Section 5.1"},{"comment":"Neither the abstract nor the experiments section mentions code or data availability. For a paper whose value depends on reproducible accuracy numbers, providing the profiling data and training scripts (or at least a clear statement of availability) would substantially strengthen the contribution.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a strong accuracy claim and includes a broad experimental evaluation, but the missing held-out split description and lack of statistical rigor are the key impediments. The issues are addressable in a revision: a detailed description of how the end-to-end held-out set was formed, a proper train/test or cross-validation protocol for the operator estimators, and some measure of variability. If the authors can show that the under-5% results hold for models genuinely outside the training distribution, the contribution would be significant. I would also suggest the editor ask for code/data release as part of the revision, given the importance of reproducibility in this line of work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look: HYMELL is a three-level hybrid cost model for LLM inference — analytical polynomial fits for GEMM/softmax/RMSNorm, small MLPs for attention/FFN blocks, and a final MLP for end-to-end latency and energy, covering dense and MoE models across prefill and decode. The layered decomposition is the genuinely new part; I have not seen this exact combination in the literature. The paper is also fairly honest about where things get hard: Table 3 shows 9.4% MAPE in the balanced compute-dominant GEMM regime, and the text attributes this to cuBLAS kernel-switching discontinuities. The cross-device A6000 check and the linear-attention ablation are good-faith attempts to show the framework is not pinned to one hardware or one attention mechanism.\n\nThe soft spots are real and, in one place, load-bearing. The central claim — under 5% error for LLaMA 3 8B, ~10% overall, and the design-space-exploration promise — depends on the end-to-end MLP generalizing beyond the roughly 1,200 profiled configurations. The paper never says whether LLaMA 3 8B, Mistral 7B, and Qwen MoE were excluded from the end-to-end training set, nor how the \"held-out\" set was constructed. If the tested models sit inside the convex hull of the training distribution, the per-model numbers are interpolation, not the architecture-level extrapolation the abstract advertises. The stress-test note is right about this.\n\nAlso: the operator-level \"analytical\" models are polynomial regressions, not first-principles derivations, and no train/test split is reported for them. Since the block- and end-to-end MLPs consume those fitted outputs, the errors at each level are not independent. There are no error bars or cross-validation variance, and the text says Figure 4 shows errors below 5% while Table 6 reports ~10% average — one of those is imprecise. No code or data is released, so the numbers cannot be independently checked. These are fixable in revision, not fatal flaws.\n\nThe paper is for ML-systems and sustainable-AI researchers who want a fast surrogate for LLM cost exploration. It deserves a serious referee: the framework is sensible, the evaluation is broad in coverage, and the limitations are mostly acknowledged. But the referee report should insist on a clear family-level split protocol, a statement about whether the named models were in the training set, and release of code/data. Without that, the headline accuracy is unverified.","headline":"A useful three-level hybrid cost model whose headline accuracy numbers currently can't be separated from interpolation because the train/test split is never pinned down.","tokens_in":13081,"tokens_out":2083,"would_cite":false,"duration_ms":19881,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HYMELL predicts LLM inference latency and energy from architecture alone, with under 5% error on LLaMA 3 8B in both prefill and decode.","keywords":["Large language models","Inference latency estimation","Energy modeling","GPU power estimation","Hybrid analytical-machine learning","Mixture-of-experts","Prefill and decode","Design space exploration"],"falsifier":"Run a held-out set of GEMM shapes whose arithmetic intensities lie just on either side of the reported thresholds (AI_low=81.92, AI_mid=166.05, AI_high=288.91 FLOPs/byte, F_launch=8.39e6 FLOPs) and compare the polynomial estimator's predictions against measured cuBLAS latency and measured energy; if the error on those boundary shapes systematically exceeds the reported 9.4% balanced-regime MAPE, the regime partition is not capturing kernel-switching discontinuities, and end-to-end predictions will inherit the distortion on workloads concentrated in those regimes.","tokens_in":12023,"feed_emoji":"⚡","tokens_out":8714,"duration_ms":68152,"temperature":0.7,"pith_summary":"HYMELL is a three-level predictor that estimates the latency and energy of large-language-model inference directly from architectural parameters, so no GPU measurement is needed at prediction time. The paper claims that combining analytical regressions for primitive GPU operations (GEMM, Softmax, RMSNorm) with lightweight MLPs at the attention and feed-forward block levels, and then an end-to-end MLP, yields roughly 10% mean absolute percentage error across dense and mixture-of-experts architectures, and under 5% error for LLaMA 3 8B in both prefill and decode. This matters because it turns architecture exploration into a fast software computation: designers can sweep hidden widths, head counts, layer counts, and MoE routing settings to find energy-efficient or latency-efficient configurations without running each candidate on hardware. The framework also reports transfer to a second GPU after re-profiling, and to a linear-attention variant after adding a block estimator.","feed_headline":"Hybrid estimator predicts LLM inference cost within 5% on LLaMA 3 8B","feed_subtitle":"Three-level hybrid of analytical and ML models hits under 5% error on LLaMA 3 8B across prefill and decode.","key_machinery":"The carrying mechanism is the three-level HYMELL hierarchy. Level 1 is a set of regime-aware analytical regressions: RMSNorm and Softmax each split into launch-bound versus memory-bound regimes by a workload threshold ($2.5\\times10^6$ for RMSNorm, $N=1024$ for Softmax), while GEMM splits into five regimes keyed by total work $F$ and arithmetic intensity $\\mathrm{AI}$, with a parallelism factor $\\mathrm{PF}=MN/K$ and chosen interaction terms in a polynomial model. Level 2 consists of small multilayer perceptrons that take the level-1 predictions plus architecture features (heads, head dimension, sequence length, $d_{\\text{model}}$, $d_{\\text{intermediate}}$, MoE expert counts) and predict attention-block and FFN-block time and energy. Level 3 is a final MLP over block predictions plus global features (layers, sequence length, batch size, inference mode) that yields end-to-end estimates. The division of labor is what carries the argument: the analytical layer tracks the dominant, hardware-regular kernels, and each MLP layer absorbs only the residual variance left by the level below it.","core_discovery":"The central claim is that inference cost for modern LLMs can be predicted accurately from architecture-level numbers by splitting the problem into three levels, each matched to the granularity where hardware behavior is regular. At level 1, analytical regressions with regime thresholds estimate runtime and energy of RMSNorm, Softmax, and GEMM; the GEMM model uses five regimes defined by total FLOPs and arithmetic intensity, with features such as a parallelism factor $MN/K$ and interaction terms to capture kernel selection. At level 2, small MLPs consume those operator-level predictions plus architecture parameters to predict attention and FFN block costs, absorbing residual overheads like reshaping, masking, activations, and MoE routing. At level 3, a final MLP combines block predictions with global parameters (layers, sequence length, batch size, prefill versus decode) to produce end-to-end estimates. The paper reports operator MAPEs below 7%, block-level MAPEs below 6%, end-to-end MAPEs of roughly 10–13% across dense and MoE configurations, under 5% for LLaMA 3 8B, and similar or better accuracy in cross-device and linear-attention ablations.","pith_inferences":["If the reported accuracy survives broader shape distributions, HYMELL could serve as a cheap surrogate for GPU profiling inside neural architecture search, letting optimizers test thousands of LLM configurations per minute rather than per measurement run.","The fixed regime thresholds are the brittle point: since cuBLAS switches kernels and tiling strategies continuously, a natural hardening is to replace the hard thresholds with a learned or interpolated mixture over regimes, or to add features that track kernel-selection boundaries.","The claim that the framework is hardware-free at prediction time still presumes a one-time profiling pass per device; on accelerators without closed-source library optimizations the regime discontinuities may be smoother, which could make the analytical layer more portable rather than less.","A direct per-configuration head-to-head against AMALI on identical models and sequence lengths, rather than comparing against AMALI's reported aggregate MAPE, would give a crisper picture of where each approach wins."],"forward_implications":["A designer can sweep model width, depth, head counts, and MoE configurations and receive latency and energy estimates for each point, making architecture-level search for efficient LLMs a software-only process.","Porting to a new GPU reduces to re-profiling the operator regressions and fine-tuning the small MLPs; the paper demonstrates this with an RTX A6000 ablation averaging about 8% MAPE.","New attention mechanisms can be folded into the framework by profiling a block-level estimator, as shown by the linear-attention ablation with sub-4% MAPE.","Because batch size is an explicit input feature, the framework claims to capture the performance and energy effects of serving optimizations such as PagedAttention and FlashAttention without changing the estimators.","The framework can be extended to heterogeneous continuous batching by evaluating the effective aggregated kernel workload rather than summing independent requests."],"supporting_citations":[{"why":"Supplies AMALI, the analytical baseline whose reported aggregate MAPE is the comparison point for the LLaMA 3 8B results.","marker":"[25]"},{"why":"Defines the LLaMA 3 8B architecture on which the under-5% prefill and decode claim is demonstrated.","marker":"[32]"},{"why":"Provides the Mistral 7B model used in the per-sequence accuracy evaluation.","marker":"[33]"},{"why":"Provides the Qwen mixture-of-experts model used to evaluate MoE accuracy.","marker":"[34]"},{"why":"Introduces the Mixture-of-Experts architecture family that HYMELL explicitly supports.","marker":"[2]"}],"fun_headline_variants":["LLM cost model hits under 5% error on LLaMA 3 8B","Hybrid analytical-ML model estimates LLM latency and energy within 5%","Three-level hybrid predicts LLM inference cost with <5% error","HYMELL: <5% error on LLaMA 3 8B inference costs","Estimate LLM inference energy and latency within 5% error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a small set of fixed regime thresholds and a fixed polynomial form can approximate how the closed-source cuBLAS library actually performs on every GEMM shape; the paper's own balanced-regime errors, up to 9.4% MAPE, mark where cuBLAS switches kernels and tiling strategies, so if those boundaries are mis-specified the operator-level inputs to the MLPs are distorted and the end-to-end accuracy is unlikely to generalize beyond the profiled shape distribution.","fun_headline_variants_meta":{"raw":{"variants":["LLM cost model hits under 5% error on LLaMA 3 8B","Hybrid analytical-ML model estimates LLM latency and energy within 5%","Three-level hybrid predicts LLM inference cost with <5% error","HYMELL: <5% error on LLaMA 3 8B inference costs","Estimate LLM inference energy and latency within 5% error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001347,"raw_usage":{"total_tokens":5512,"prompt_tokens":1028,"completion_tokens":4484,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":4378}},"tokens_in":644,"tokens_out":4484,"duration_ms":28904,"temperature":1.0,"reasoning_tokens":4378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:50:53.306337+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a held-out set of GEMM shapes whose arithmetic intensities lie just on either side of the reported thresholds (AI_low=81.92, AI_mid=166.05, AI_high=288.91 FLOPs/byte, F_launch=8.39e6 FLOPs) and compare the polynomial estimator's predictions against measured cuBLAS latency and measured energy; if the error on those boundary shapes systematically exceeds the reported 9.4% balanced-regime MAPE, the regime partition is not capturing kernel-switching discontinuities, and end-to-end predictions will inherit the distortion on workloads concentrated in those regimes.","supporting_citations":[{"cited_title":"Amali: An analytical model for accurately modeling llm inference on modern gpus","cited_arxiv_id":null,"evidence_quote":"Supplies AMALI, the analytical baseline whose reported aggregate MAPE is the comparison point for the LLaMA 3 8B results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Mistral 7B model used in the per-sequence accuracy evaluation."}],"review_version":1}